Methods and system for improved quality inspection

ABSTRACT

A method of performing automated object inspection includes obtaining a plurality of test images. For each of the plurality of test images, the method includes quantitatively determining a respective transformation from a predefined contour of the reference marker in the predefined common layout to a respective anchor contour corresponding to the reference marker as captured in the test image; and applying an inverse of the respective transformation to at least a portion of the test image to obtain a respective regularized version of the test image, such that the reference markers captured in the respective regularized versions of the plurality of test images share an image-independent location and orientation. The method further includes performing independent automated inspection on each of two or more sub-portions of the respective regularized versions of the test images.

TECHNICAL FIELD

This relates generally to quality inspection, including but not limitedto visually inspecting products in a product assembly line or during apost-production quality assurance process.

BACKGROUND

Manufacturers implement various quality control measures to reduce theamount of defective products that enter the stream of commerce. Somequality control measures involve human workers and/or devices visualinspecting products. However, current visual inspection techniques facelow accuracy and low efficiency for complex object inspection and alsodo not provide convenient user interactions with the system. As such,challenges exist in identifying product defects for a complex productincluding multiple components and multiple types of defects.

SUMMARY

Accordingly, there is a need for methods and systems for performingefficient and accurate inspection of a complex product includingmultiple component, and each component may include different types ofdefects. In this way, a manufacturer is able to further reduce an amountof defective products entering the stream of commerce while alsodetecting the defects in such a manner that manufacturing down time isreduced and defect detection accuracy is improved.

In some embodiments, detecting and using location and orientation ofreference markers to standardize captured images before performingvisual inspection of the components of the composite object or theassessor pack has the advantage of high degree of an automatic process,thus more efficient and accurate. Moreover, the system can be easilymodified, e.g., by updating the blueprint, or constantly performingmachine learning to incorporate any updates to the model, and timelyidentifying defects that have been intentionally or inadvertentlyintroduced to the assembly lines.

In some embodiments, by using a pre-segmentation process, the accuracyof the inspection system can be improved (e.g., from 80%-90% to 99.9%).The users can do the segmentation manually by select the segmented areain the standard image. The users select an obvious feature in the imageand the system will use this feature for standardization. Each of theimage will be orientated and scaled, so the detecting area in each ofthe image will be in the same size and shape. The users define thepre-segmentation area in a standard image. Each of the followingstandardized image will be segmented in the same way. Each of thesegmented area will be trained into a single model. The system willdetect object in each of the segmented area instead of in the wholeimage. By using these methods, the detecting accuracy will be increasedgreatly.

Furthermore, by manually selected or auto-extracted multiple informationchannels of an image, the detection efficiency and the accuracy can beimproved. The system can support multiple channels input, such as RGBD.It also can extend to ultrasonic, inferred or laser. Each channel has aweight in the result of detection. The weight can be pre-defined orauto-extracted by learning in a plurality of samples or during along-term operation.

The system supports a cloud computing framework connecting the edgedevices and the server system, thus the system can handle multiple taskssimultaneously. By using a server system that leverages cloud computingpower, the inspection cost can be reduced at assembly line. The systemfurther improves efficiency with high computing power. Additionally, thesystem has the flexibility of using more training data from multipleassembly lines to train the model to adapt the model to differentsituations, or using training data from a certain assembly line toimprove accuracy to target at a specific assembly line.

The current system and method can identify a defective product beforeentering the stream of commerce. Moreover, the cause of the defect canbe identified and remedied, thereby reducing the downtime of the system.Moreover, by isolating the individual components, and the individualinformation channels, the training of a model can be faster, and themodel can be more sensitive (e.g., due to less interference ofirrelevant information from other objects or channels), producing higherinspection accuracy.

In some embodiments, a method of performing automated object inspection,comprising: at a computing device (e.g., server system 152, FIG. 1B;server system 200, FIG. 2) having one or more processors and memory,obtaining a plurality of test images, each test image capturing arespective one of a plurality of composite objects that have beenassembled in accordance with a predefined common layout for theplurality of composite objects. The predefined common layout specifiesrespective positions for two or more components of a respectivecomposite object, and a predefined location and orientation for areference marker, in the respective composite object. Each test imagecaptures respective portions of a corresponding composite object thatcontain the two or more components and the reference marker, andcaptures at least a portion of a surrounding environment of thecorresponding composite object. For each of the plurality of testimages, the method comprises quantitatively determining a respectivetransformation from a predefined contour of the reference marker in thepredefined common layout to a respective anchor contour corresponding tothe reference marker as captured in the test image. The method furthercomprises applying an inverse of the respective transformation to atleast a portion of the test image to obtain a respective regularizedversion of the test image, such that the reference markers captured inthe respective regularized versions of the plurality of test imagesshare an image-independent location and orientation. The method alsocomprises performing independent automated inspection on each of two ormore sub-portions of the respective regularized versions of the testimages. Each sub-portion of the respective regularized versions of thetest images corresponds to a respective one of the two or morecomponents of the respective composite object captured in the testimage.

In accordance with some embodiments, a system includes one or moreprocessors, and memory storing one or more programs; the one or moreprograms are configured to be executed by the one or more processors andthe one or more programs include instructions for performing or causingperformance of the operations of any of the methods described herein. Inaccordance with some embodiments, a non-transitory computer readablestorage medium has stored therein instructions, which, when executed byan electronic device, cause the device to perform or cause performanceof the operations of any of the methods described herein.

In some embodiments, a method of performing automated object inspection,comprising: at a computing device (e.g., server system 152, FIG. 1B;server system 200, FIG. 2) having one or more processors and memory,obtaining a plurality of test images, each test image capturing arespective one of a plurality of composite objects comprising two ormore components. For each of the plurality of test images, the methodincludes performing independent object inspection on each of two or moresub-portions of the test image. Each sub-portion of the test imagecorresponds to a respective one of the two or more components of therespective composite object captured in the test image. Performing theindependent object inspection includes segmenting the test image into atleast a first sub-portion of the test image and a second sub-portion ofthe test image, the first sub-portion of the test image containing afirst component and not containing a second component of the two or morecomponents, and the second sub-portion of the test image containing thesecond component and not containing the first component of the two ormore components. The method comprises performing object inspection onthe first sub-portion of the test image using a first subset ofinformation channels of the test image and a first model trained on afirst set of training images containing the first component and notcontaining the second component. The method further comprises performingobject inspection on the second sub-portion of the test image using asecond subset of information channels of the test image that aredistinct from the first subset of information channels, and a secondmodel trained on a second set of training images containing the secondcomponent and not containing the first component.

In accordance with some embodiments, a system includes one or moreprocessors, and memory storing one or more programs; the one or moreprograms are configured to be executed by the one or more processors andthe one or more programs include instructions for performing or causingperformance of the operations of any of the methods described herein. Inaccordance with some embodiments, a non-transitory computer readablestorage medium has stored therein instructions, which, when executed byan electronic device, cause the device to perform or cause performanceof the operations of any of the methods described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described embodiments,reference should be made to the Description of Embodiments below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures and specification.

FIG. 1A is an exemplary arrangement of a plurality of edge devices on anassembly line, in accordance with some embodiments.

FIG. 1B is a block diagram illustrating an exemplary networkarchitecture of a system for performing visual inspection of an assemblyline, in accordance with some embodiments.

FIG. 2 is a block diagram illustrating an exemplary server system, inaccordance with some embodiments.

FIG. 3A is an exemplary image illustrating a top-down view showing anexemplary view of a composite product placed on the assembly line, inaccordance with some embodiments.

FIGS. 3B-3E are exemplary images illustrating various defects of acomposite object under inspection on an assembly line, in accordancewith some embodiments.

FIGS. 4A-4F illustrate a process of performing object detection andreference marker alignment to a product placed on an assembly line forinspection, in accordance with some embodiments.

FIGS. 4G-4J are flow diagrams illustrating four main processes of objectdetection and alignment as illustrated in FIGS. 4A-4F, in accordancewith some embodiments.

FIG. 5 is an exemplary image of a product including a plurality ofinformation channels, in accordance with some embodiments.

FIG. 6A illustrates a process of preparing one or more maskscorresponding to one or more components of a product for visualinspection, in accordance with some embodiments.

FIG. 6B is a flow diagram illustrating a method of preparing one or moremasks corresponding to one or more components of a product for visualinspection, in accordance with some embodiments.

FIG. 6C is a flow diagram illustrating a method of preparing one or moremasks corresponding to one or more components of a product for visualinspection, in accordance with some embodiments.

FIG. 7A illustrates a process of training one or more models for one ormore components of a product for visual inspection, in accordance withsome embodiments.

FIG. 7B is a flow diagram illustrating a method of training one or moremodels for one or more components of a product for visual inspection, inaccordance with some embodiments.

FIG. 7C is a flow diagram illustrating a method of training one or moremodels for one or more components of a product for visual inspection, inaccordance with some embodiments.

FIGS. 8A-8B illustrate a process of performing product inspection usingmodels, in accordance with some embodiments.

FIG. 8C is a flow diagram illustrating a method of performing productinspection using models, in accordance with some embodiments.

FIG. 8D is a flow diagram illustrating a method of performing productinspection using models, in accordance with some embodiments.

FIG. 9A is a flow diagram illustrating a method of training one or moremodels for performing visual inspection of a product, in accordance withsome embodiments.

FIG. 9B is a flow diagram illustrating a method of performing productinspection using models, in accordance with some embodiments.

FIG. 10 is an example user interface for a user to interact withtraining models and performing inspection of a product using a mobiledevice with a touch screen, in accordance with some embodiments.

FIG. 11A is a flow diagram illustrating a method of performing automatedobject inspection on an assembly line, in accordance with someembodiments.

FIG. 11B is another flow diagram illustrating a method of performingautomated object inspection on an assembly line, in accordance with someembodiments.

DESCRIPTION OF EMBODIMENTS

Reference will now be made to embodiments, examples of which areillustrated in the accompanying drawings. In the following description,numerous specific details are set forth in order to provide anunderstanding of the various described embodiments. However, it will beapparent to one of ordinary skill in the art that the various describedembodiments may be practiced without these specific details. In otherinstances, well-known methods, procedures, components, circuits, andnetworks have not been described in detail so as not to unnecessarilyobscure aspects of the embodiments.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are used onlyto distinguish one element from another. For example, a first edgedevice could be termed a second edge device, and, similarly, a secondedge device could be termed a first edge device, without departing fromthe scope of the various described embodiments. The first edge deviceand the second edge device are both edge devices, but they are not thesame edge devices.

The terminology used in the description of the various describedembodiments herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used in thedescription of the various described embodiments and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting”or “in accordance with a determination that,” depending on the context.Similarly, the phrase “if it is determined” or “if [a stated conditionor event] is detected” is, optionally, construed to mean “upondetermining” or “in response to determining” or “upon detecting [thestated condition or event]” or “in response to detecting [the statedcondition or event]” or “in accordance with a determination that [astated condition or event] is detected,” depending on the context.

As used herein, the term “exemplary” is used in the sense of “serving asan example, instance, or illustration” and not in the sense of“representing the best of its kind.”

FIG. 1A is an exemplary arrangement of a plurality of edge devices 104on an assembly line 100, in accordance with some embodiments. Theassembly line 100 includes a work surface 120 (or a work platform). Insome embodiments, the work surface 120 conveys products through theassembly line 100. In some embodiments, the work surface 120 may movecontinuously along a certain direction to convey one or more products122 for inspection on the assembly line 100. Alternatively, theembodiments described herein apply equally to packaging, assembly, ormanufacturing processes. In some embodiments, the work surface 120 isstationary. For example, the work surface 120 may be a surface of athree-dimensional printer, a computer numerical control (CNC) millingmachine, or any other non-assembly line based manufacturing and/orinspection process. In some embodiments, a product 122 is a compositeproduct. In some embodiments, a composite product 122 includes aplurality of components (also referred to as a multi-component product)that are coupled to each other in a predefined design. For example, thecomposite product 122 is an electric fan including a head coupled to aplatform. In some embodiments, a composite product 122 is an accessorypack including multiple objects (e.g., the accessory pack 122 includingmultiple objects 140, 142, and 144 and a reference marker 146, FIG. 1A)that have been assembled in accordance with a predefined common layout.For example, the composite product 122 is an accessory box packed withvarious accessories, such as electric cables, battery pack, instructionmanual, and a tool kit.

In some embodiments, the assembly line 100 includes one or more edgedevices 104-1, 104-2, 104-3, . . . 104-n located over the work surface120 towards the product 122 that is placed on the work surface 120 forinspection. In some embodiments, the edge devices 104-1, 104-2, 104-3, .. . 104-n are installed on respective robotic arms 102-1 . . . 102-mthat are configured to adjust the locations and/or orientations of therespective edge devices to obtain inspection data (e.g., images or othertypes of sensory data) of the product 122. In some embodiments, the edgedevices 104-1, 104-2, 104-3, . . . 104-n include various sensorsconfigured to collect data related to defects of the product 122 on theassembly line 100 and perform an automated object inspection (e.g.,including steps as discussed with reference to FIGS. 11A-11B) of theproduct 122 by analyzing the data collected during the inspection. Insome other embodiments, the edge devices 104-1, 104-2, 104-3, . . .104-n collect data related to defects of the product 122 and thentransmit the inspection data to a server system (e.g., server system152, FIG. 1B) to perform the automated object inspection (e.g., asdiscussed with reference to FIGS. 11A-11B).

In some embodiments, to inspect the product 122, each edge device 104-1,104-2, 104-3, . . . 104-n is a type of sensor device configured tocapture a certain characteristic data (corresponding to a certain typeof sensory data) of the product 122. For example, the first edge device104-1 is a camera (e.g., a color camera, a black/white camera, or aninfrared camera) configured to capture an image of the product 122. Forexample, a color camera 104-1 includes sensors and color filtersconfigured to obtain different primary color information (e.g., threeRGB primary colors) of a captured image. The different primary colorinformation of a captured image can be used to train separate modelscorresponding to respective primary colors for visual inspection toimprove accuracy. In some embodiments, the second edge device 104-2 is adepth sensor (e.g., a time-of-flight camera or a range imaging camera,such as a LiDAR system) that resolves distance by measuringtime-of-flight of light signal between the edge device 104-2 and asurface of an object of the product 122. Alternatively, the depth sensorcan be an ultrasound sensor or any other suitable sensors. In someembodiments, the third edge device 104-3 is weighing sensor configuredto obtain a weight of the product 122. The weighing sensor may belocated within or below the work surface 120. In some embodiments, oneor more other edge devices 104-4 . . . 104-n include various other typesof sensors (e.g., touch-sensors, vibration sensors, sound sensors,haptic sensors, temperature sensors, light sensors, movement sensors,etc.) for collecting information about the state of the compositeproduct 122 and the assembly line 100. In some embodiments, the firstedge device 104-1 (e.g., a camera, a depth camera, a video camera, or athree-dimensional camera, etc.) is used to identify a first type ofproduct defect (e.g., surface defects), and the second edge device 104-2(e.g., an infrared camera, an X-ray camera, etc.) is used to identify asecond type of product defect (e.g., internal defects).

In some embodiments, the edge devices are positioned at fixed locationsaround the assembly lines or inspection stations. In some embodiments,the edge devices are actuated and positioned by one or more robotic arms(e.g., a robotic arm 102) around the assembly lines or inspectionstations. In some embodiments, a certain edge device, e.g., the camera104-1, is configured to capture images of the product 122 from the sameposition (e.g., the same distance to the work surface 120, the sameangle towards a standard placing position to position the product 122 onthe work surface). Although not shown, each of the edge devices 104-1,104-2, 104-3, . . . 104-n is supported by a support apparatus or ismounted in some other manner. These supports or mounts have been removedfor ease of illustration.

In some embodiments, the edge devices 104-1, 104-2, 104-3, . . . 104-nsend the data collected during inspection of the product 122 to theserver system 152 (FIG. 1B) to leverage the processing power of theserver system 152. In some embodiments, the edge devices 104-1, 104-2,104-3, . . . 104-n send the data to the server system 152 at apredetermined interval, after identifying a product defect, or afteridentifying some other irregularity. The server system processes thedata received from the edge devices 104-1, 104-2, 104-3, . . . 104-n topotentially record and report the product defect. In some embodiments,in processing the data, the server system analyzes individual uniquecharacteristic of the product received from each of the edge devices104-1, 104-2, 104-3, . . . 104-n. For example, the server system trainsa respective model using training data obtained from a correspondingedge device and uses the respective model to analyze the test dataobtained from the corresponding edge device to determine whether theproduct contains a certain defect. In some embodiments, the serversystem updates the respective models based on the identified defectsdetected during testing. In some embodiments, the server system sendsthe respective models to the edge devices 104-1, 104-2, 104-3, . . .104-n to have the edge devices perform defect inspection after receivingthe inspection data. In this way, the production line 100 implementsmachine learning allowing the edge devices to dynamically identify andaccount for product defects (e.g., flag the defective product so that itdoes not enter the stream of commerce).

FIG. 1B is a block diagram illustrating an exemplary networkarchitecture 150 of a system for performing visual inspection of anassembly line (e.g., the assembly line 100, FIG. 1A) in accordance withsome embodiments. The network architecture 150 includes a number of edgedevices 104-1, 104-2, . . . 104-n communicably connected to a serversystem 152 by one or more networks 106. In some embodiments, the networkarchitecture 150 further comprises a mobile device 156 that can becarried by mobile quality control personnel. For example, the personnelcould use the mobile device 156 to capture an image or capture any othertypes of data of the product 122, and the captured image and/or data canbe processed by the mobile device 156 or transmitted to the serversystem 152 to detect whether the product includes any defects.

In some embodiments, the edge devices 104-1, 104-2, . . . 104-n areelectronic devices that can communicate with the server system 152, eachother, and other devices. In some embodiments, the server system 152 isa single computing device such as a computer server, while in otherembodiments, the server system 152 is implemented by multiple computingdevices working together to perform the actions of a server system(e.g., cloud computing). In some embodiments, the network 106 is apublic communication network (e.g., the Internet or a cellular datanetwork), a private communications network (e.g., private LAN or leasedlines), or a combination of such communication networks.

The edge devices 104-1, 104-2, . . . 104-n are used to inspect (e.g.,monitor) the assembly line 100 for product defects. In some embodiments,the edge devices 104-1, 104-2, . . . 104-n monitor an operation of theassembly line 100 (e.g., monitor movements of an operator). In someembodiments, the edge devices 104-1, 104-2, . . . 104-n monitor anoperation's effect on a product (e.g., perform quality control). Theedge devices 104-1, 104-2, . . . 104-n capture unique perspectives ofthe operation (e.g., capture unique perspectives of a product and/orunique perspectives of an operator performing an operation). To do this,each of the edge devices 104-1, 104-2, . . . 104-n includes one or moresensor devices, such as a camera, an infrared camera, an X-ray camera, adepth camera, etc. The goal being that the edge devices 104-1, 104-2, .. . 104-n can identify product defects (or collect data that can be usedto identify product defects) and report to the system to improve theproduction process. In some embodiments, the edge devices 104-1, 104-2,. . . 104-n send the captured data to the server system 104.

In some embodiments, the network architecture 100 may also includethird-party servers (not shown). In some embodiments, third-partyservers are associated with third-party service providers that provideadditional data the server system 152 (e.g., weather data and personneldata).

FIG. 2 is a block diagram illustrating an exemplary server system 200,in accordance with some embodiments. In some embodiments, the serversystem 200 is an example of a server system 154 (FIG. 1B). The serversystem 200 typically includes one or more processing units (processorsor cores) 202, one or more network or other communications interfaces204, memory 206, and one or more communication buses 208 forinterconnecting these components. The communication buses 208 optionallyinclude circuitry (sometimes called a chipset) that interconnects andcontrols communications between system components. The server system 200optionally includes a user interface (not shown). The user interface, ifprovided, may include a display device and optionally includes inputssuch as a keyboard, mouse, trackpad, and/or input buttons. Alternativelyor in addition, the display device includes a touch-sensitive surface,in which case the display is a touch-sensitive display.

Memory 206 includes high-speed random access memory, such as DRAM, SRAM,DDR RAM, or other random access solid state memory devices; and mayinclude non-volatile memory, such as one or more magnetic disk storagedevices, optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. Memory 206 may optionallyinclude one or more storage devices remotely located from theprocessor(s) 202. Memory 206, or alternately the non-volatile memorydevice(s) within memory 206, includes a non-transitory computer readablestorage medium. In some embodiments, memory 206 or the computer readablestorage medium of memory 206 stores the following programs, modules, anddata structures, or a subset or superset thereof:

-   -   an operating system 210 that includes procedures for handling        various basic system services and for performing        hardware-dependent tasks;    -   a network communication module 212 that is used for connecting        the server system 200 to other computers (e.g., edge devices        104-1, 104-2, . . . 104-n, and/or third party servers) via the        one or more communication network interfaces 204 (wired or        wireless) and one or more communication networks 106 (FIG. 1B),        such as the Internet, cellular telephone networks, mobile data        networks, other wide area networks, local area networks,        metropolitan area networks, and so on;    -   a user input processing module 214 that is used for receiving        and processing user input;    -   a mask creation module 216 for creating masks used for training        and inspection processes;    -   an imaging processing module 218 for receiving captured images        from the camera, compressing the images to low-resolution        images, applying masks, and cropping respective components of        the images;    -   a transformation determination module 220, including        -   a contour extracting module 222 for extracting contours of            patterns of an image;        -   an anchor identifying module 224 for identifying an anchor            (e.g., a reference marker) of an image;        -   an affine transformation determining module 226 for            determining an affine matrix between the anchor of a            standardized image and the anchor of a test image; and        -   a histogram analysis module 228 for calculating histogram            including translation offset and angular shift from a            standardized image to a test image;    -   a test image regularization module 230 for applying an inverse        of the affine transformation to standardize a test image;    -   a model training module 232 for training models using machine        learning/deep leaning processes;    -   a segmentation module 234 for performing pre-segmentation of a        test image to identify respective components of an image;    -   an object inspection module 236 for performing inspection of a        test image using models; and    -   a server database 250 for storing data associated with the        server system, such as:        -   one or more masks 252;        -   one or more histograms 254;        -   one or more models 256;        -   one or more predefined common layout 258;        -   one or more criteria and thresholds 260; and        -   content 262.

The content 262 can include data received from the edge devices, such asunique perspectives captured by the edge devices. In addition, thecontent 262 can include models and views generated by the server system(or models and views received from one or more edge devices). In someembodiments, the content 262 includes text (e.g., ASCII, SGML, HTML),images (e.g., jpeg, tif and gif), graphics (e.g., vector-based orbitmap), audio, video (e.g., mpeg), other multimedia, and/orcombinations thereof.

The server database 250 stores data associated with the server system200 in one or more types of databases, such as text, graph, dimensional,flat, hierarchical, network, object-oriented, relational, and/or XML,databases.

In some embodiments, the server system 200 stores in memory a graph ofthe edge devices. For example, the graph identifies each edge device ona particular production line and connections between each edge device.The connections may include a position of the edge device, anorientation of the edge device, neighboring edge devices, etc. Bymaintaining the graph, the server system 200 is able to determine howunique perspectives relate to one another.

FIG. 3A is an exemplary image illustrating a top-down view 300 showingan exemplary view of a composite product (e.g., the product 122) placedon the assembly line, in accordance with some embodiments. In someembodiments, the top-down view 300 includes a camera view 302 that iscaptured by the edge device 104-1 as shown in FIG. 1A, when the product122 is placed on the work surface 120 of the assembly line 100. In someembodiments, the product 122 is an accessory pack including a pluralityof objects 140, 142, and 144, and a reference marker 146. In someembodiments, the reference marker 146 is a printed mark, hole, orprotrusion, that is predesigned to have an asymmetric pattern and has afixed location and orientation on the product or the accessory box, suchthat the orientation and lateral position of the product or theaccessory box can be deduced accurately once the location andorientation of the reference marker 146 is determined.

FIGS. 3B-3E are exemplary images illustrating various defects of thecomposite product 122 under inspection on the assembly line 100, inaccordance with some embodiments. In some embodiments as shown in FIG.3B, a defect to the product 122 exists when an object 144 that is placedupside down 312 in the product 122 (e.g., an accessory pack). In someembodiments, because the product 122 may be dropped and/or pushed intothe camera view 302 after they are assembled, they may be of slightlydifferent lateral positions and rotational angles on the table, even ifthey are all laid flat on the table. For example, as shown in FIG. 3C,the product 122 includes a defect when the product is misaligned 322,e.g., placed tilted relative to a standard location and orientation ofthe product 122 on the assembly line 100. In some embodiments as shownin FIG. 3D, the product 122 has a defect when an object (e.g., theobject 140 as shown in FIG. 3A) is missing 332. In some embodiments asshown in FIG. 3E, one or more objects, e.g., the object 142 of theproduct 122 includes one or more surface defects 342, such as scratches,blemishes, visible cracks, abrasion, corrosion, and/or debris. One ormore edge devices 104-1 . . . 104-n are used to detect one or moredefects associated with the product 122 using the system and method asdiscussed herein so as to identify a defective product before enteringthe stream of commerce. The defects can be identified efficiently andtimely, thereby reducing the downtime and cost for product inspection.

FIGS. 4A-4F illustrates a process of performing object detection andreference marker alignment to a product placed on an assembly line forinspection, in accordance with some embodiments. As shown in FIG. 4A,when the product 122 is placed on the assembly line, the product 122 maybe shifted and/or rotated from a standard position 412 of the product122 with the reference marker at position 402 to a misaligned location414 of the product 122 with the reference marker at position 404. Insome embodiments, the product 122 includes a reference marker (e.g.,reference marker 146, FIG. 3A) at a fixed location and orientationrelative to the product 122. The reference marker is used to determinethe orientation and lateral position of the product 122 at themisaligned location 414 relative to the standard position 412.

FIGS. 4G-4J are flow diagrams illustrating four main processes of objectdetection and alignment as illustrated in FIGS. 4A-4F, in accordancewith some embodiments. The first process 430 as illustrated in FIG. 4Gis related to objection detection and alignment performed to a standardsample image. The second process 450 as illustrated in FIG. 4H isrelated to an anchor detection performed to a test image captured on theassembly line. The third process 470 as illustrated in FIG. 4I isrelated to a first method of detecting key points on the anchor contourand determining an affine matrix based on the detected key points of thetest image. The fourth process 480 as illustrated in FIG. 4J is directedto a second method of key points detection on the anchor contour andaffine matrix calculation. In some embodiments, the steps of theprocesses illustrated in FIGS. 4G-4J are performed by the server system152. In some embodiments, one or more steps of the processes illustratedin FIGS. 4G-4J are performed by one or more edge devices 104 and/or themobile device 156.

In performing the process 430 in FIG. 4G, the server system receives(432) a standard sample image (e.g., camera view 302 of FIG. 3A)captured by the edge device 104-1. In some embodiments, the standardsample image shows one or more objects placed or assembled in apredefined common layout. In some embodiments, the predefined commonlayout is a blueprint of the design of the product 122. In some otherembodiments, the common layout is an image captured by the cameralocated over the assembly line 100 of a standard example of the product122 placed at a standard position 412 (e.g., upright and squarelyaligned in the center of the table) on the assembly line.

The server system then selects (434) an anchor area (not shown) whichencloses the reference marker 402 on the standard sample image. In someembodiments, the server system selects the anchor area using an imagerecognition technique or a machine learning process. In someembodiments, the anchor area is selected by a user's manual input. Forexample, when the standard sample image from the camera view 302 isdisplayed on the user's mobile device 156, the user can use one or moreinput devices (e.g., touch screen, keypad, stylus, etc.) of the mobiledevice 156 to select the anchor area.

The server system transforms (436) the anchor area in the standardsample image into a binary picture using a thresholding process. Forexample, an intensity value of the anchor area at each pixel is comparedwith a predetermined threshold value. One or more pixels in the anchorarea are replaced with black pixels in accordance with determinationsthat the intensity values of these pixels are less than thepredetermined threshold value. On the other hand, one or more pixels inthe anchor area are replaced with white pixels in accordance withdeterminations that the intensity values of these pixels are greaterthan the predetermined threshold value. The thresholding process can beperformed automatically by the system.

The server system further extracts (438) an anchor contour (e.g., anchorcontour 416, FIG. 4B) of the reference marker at position 402 in thestandard sample image 412 using any suitable imaging processingtechnique. After determining the anchor contour 416, the server systemalso calculates (440) the acreage (A) of the anchor (e.g., the areaenclosed by the anchor contour of the reference marker). Additionally,the server system determines and saves (442) shape features, such as thenumber of local maximum distances from the center point 420 of theanchor contour 416 to each point along the anchor contour 416. In someembodiments as illustrated in FIG. 4C, the server system extracts (444)a minimal enclosing rectangle 406 that encloses the anchor contour ofthe reference marker. The server system further extracts (444) foursequential points of the rectangle 406 as shown in FIG. 4C. The serversystem then calculates (446) a center point 420 of the anchor as shownin FIG. 4D. Additionally, the server system calculates (448) a histogram428 as shown in FIG. 4E of distances D′(0′) between the point 420 of theanchor contour 416 and each of a sequence of n equally spaced points(including point i 422, 4D) on the anchor contour (e.g., adjacent pointsare 360/n degrees apart or 360/m degrees apart with m>n and n pointssequentially located to each other). As shown in FIG. 4D, 0′ indicatesan angle between a line connecting the center point 420 and point i 422,and a horizontal line.

For example, as shown in FIG. 4E, the x axis represents an angle between(1) a line connecting a point i 422 on the anchor contour and the centerpoint 420 and (2) a horizontal line. The point i 422 is selected to be

P _(traini)=(P _(txi) ,P _(tyi))  (1)

The center point 420 is

CP _(train)=(C _(tx) ,C _(ty))  (2)

The distance between the point i 422 and the center point 420 is

${D^{\prime}\left( \theta^{\prime} \right)} = \sqrt{\left( {P_{txi} - C_{tx}} \right)^{2} + \left( {P_{tyi} - C_{ty}} \right)^{2}}$$\theta^{\prime} = {\arctan \frac{\left( {P_{tyi} - C_{ty}} \right)}{\left( {P_{txi} - C_{tx}} \right)}}$

Proceeding to the second process 450 in FIG. 4H, the server systemreceives (452) a test image captured by the edge device 104-1 from theassembly line. In some embodiments as show in FIG. 4A, the test imageincludes the product 122 placed in the misaligned position 414 with thereference marker at position 404. The server system transforms (454) thetest image into a binary picture. The server system then extracts (456)one or more contours (e.g., contours 418, 419) of one or more imagepatterns recognized from the binary picture. In some embodiments, theserver system maintains (456) a contour list including n (n=1, 2, . . .) recognized contours from the binary picture. After extracting all thecontours from the binary image, the server system then examines each ofthe n recognized contours to identify the contour for the referencemarker 404 (FIG. 4A). The server system starts from a first contour bysetting (458) i as 1, and calculates (460) an acreage (Ai) of an areaenclosed by contour i from the contour list. The server system thendetermines (462) whether a difference between the acreage (Ai) ofcontour i in the binary image of the test image and the acreage (A) ofthe anchor contour in the binary image of the standard sample image(e.g., calculated in step 440 of FIG. 4G) is no greater than apredetermined threshold value (e.g., a predetermined threshold ratio dtimes A: d*A):

|A _(i) −A|≤d×A

In accordance with a determination that the difference is greater thanthe predetermined threshold value, the server system increases (467) thecounter i by 1 to examine the next pattern contour. In accordance with adetermination that the difference is less than or equal to thepredetermined threshold value, the server system proceeds to calculate(464) a shape feature Si of the contour i. In some embodiments asdiscussed herein, the shape feature Si is calculated based on the numberof local maximum distances from the center point of the contour i toeach point along the contour i (e.g., the number represents the numberof apexes of the shape). The server system then determines (466) thecloseness between the shape feature Si and the shape feature S of theanchor contour in the binary image of the standard sample image (e.g.,determined in step 442 of FIG. 4G). For example, the difference betweenthe shape feature Si and the shape feature S is compared with apredetermined threshold value t to examine whether:

|S _(i) −S|≤t

In accordance with a determination that the difference is less than orequal to t, the server system determines (468) that contour i is thetarget contour for the anchor (e.g., the contour 418 in FIG. 4Bcorresponds to the reference mark or the anchor 404 in FIG. 4A). Inaccordance with a determination that the difference is greater than t,the server system increases (467) the counter i by 1 to examine the nextpattern contour.

Proceeding to the third process 470 in FIG. 4I, the server systemidentifies (472) the target contour 418 from the test image as discussedwith reference to FIG. 4H. The server system then determines (474) aminimum enclosing rectangle 408 (FIG. 4C) that encloses the contour 418.The server system determines (476) four sequential points of therectangle 408 as shown in FIG. 4C. The server system then calculates(478) an affine matrix based on the four points (of rectangle 408) ofthe test image and the four points (of rectangle 406) of the trainingsample image as illustrated in FIG. 4C.

Proceeding to the fourth process 480 in FIG. 4J, the server systemidentifies (482) the target contour 418 from the test image as discussedwith reference to FIG. 4H. The server system then determines (484) acenter point (e.g., center point 424 of FIG. D) of the target contour418. The server system then calculates (486) a histogram 429 as shown inFIG. 4F of distances Di(θ) between the center point 424 and each of aset of points on the target contour 418. The server system then selects(488) x adjacent and equally spaced points from the target contour 418of the test image (e.g., adjacent points are 360/n degrees apart, n≥).The server system selects (490) corresponding x points from the anchorcontour of the training sample image, set θ=k*₃. The server systemfurther calculates (492) distances between the x points and the centerpoint 420 of the anchor contour in the training sample image. The serversystem then compares (494) the histogram 428 (FIG. 4E) of the trainingsample image and the histogram 429 (FIG. 4F) of the test image toidentify the minimal distance (Δθ), and point K on the anchor of thetest image that corresponds to the point O (θ =0) on the anchor of thestandard sample image.

For example, as shown in FIG. 4F, the x axis represents an angle between(1) a line connecting a point i 426 on the anchor contour and the centerpoint 424 and (2) a horizontal line. The point i 426 is selected to be

P _(traini)=(P _(txi) ,P _(tyi))  (1)

The center point 420 is

CP _(train)=(C _(tx) ,C _(ty))  (2)

The distance between the point i 422 and the center point 420 is

${D(\theta)} = \sqrt{\left( {P_{texi} - C_{tex}} \right)^{2} + \left( {P_{teyi} - C_{tey}} \right)^{2}}$$\theta = {\arctan \frac{\left( {P_{teyi} - C_{tey}} \right)}{\left( {P_{texi} - C_{tex}} \right)}}$

In some embodiments, there are three steps to find the key points:

Step 1: the system selects n points from the target anchor from the testpicture that are adjacent and equally spaced from each other. The anglebetween two adjacent points is set as ζ. Then the system can get apoints list:

     P_(tj) = (P_(txj), P_(tyj))  (P_(tj) ∈ P_(train), j ∈ [1, n]), and${{\theta_{j} - \theta_{j - 1}} = {{\varsigma \left( {\theta = {\arctan \frac{\left( {P_{teyi} - C_{tey}} \right)}{\left( {P_{texi} - C_{tex}} \right)}}} \right)}\mspace{14mu} \left( {{P_{tj} \in P_{i\mspace{14mu} {test}}},{j \in \left\lbrack {1,n} \right\rbrack}} \right)}},{{{and}\mspace{14mu} \theta_{0}} = 0.}$

Then the system can get n distance values.

Step 2: the system selects n points from the sample image, the anglebetween adjacent points is the same as ζ. Then the system can get apoints list:

P_(tj) = (P_(txj), P_(tyj))  (P_(tj) ∈ P_(test), j ∈ [1, n]), and${{\theta_{j} - \theta_{j - 1}} = {{\varsigma \left( {\theta = {\arctan \frac{\left( {P_{teyi} - C_{tey}} \right)}{\left( {P_{texi} - C_{tex}} \right)}}} \right)}\mspace{14mu} \left( {{P_{tj} \in P_{i\mspace{14mu} {test}}},{j \in \left\lbrack {1,n} \right\rbrack}} \right)}},$

for initiation station, set θ₀=0. Then the system can get n distancevalues.

Step 3: the system gets the distance function between the above twopoint lists. Set the function as

DD _(k)(j)=Σ(D _(i)(θ_(j))−D _(k)(θ_(j)),

Then set θ₀=k×ζ, then the system will find the minimal DD_(k)(j) and thek points in the training sample correspond to the training sample pointwhose θ=0. In some embodiments, the number of points included in thepoint lists are reduced to save the calculation time and improve theefficiency of the system. In some other embodiments, one or morelimitations can be added to the calculation. For example, if the objectcan only rotate within a range between −β to β, the rotation angle foranchor detection can also be limited to a range between −β to β, thenthe system needs only calculate the distance of the points between −β toβ, so as to save the processing time and improve efficiency.

FIG. 5 is an exemplary image 500 of the product 122 including aplurality of information channels 510, 520, and 530, in accordance withsome embodiments. In some embodiments, the image 500 is captured by theedge device 104-1, such as a color camera. The image 500 may include aplurality of information channels, such as a red color channel (R), agreen color channel (G), and a blue color channel (B). In someembodiments, the image 500 may further include depth information (D)which is captured by a depth camera. In some embodiments, asingle-channel image, e.g., R channel image 510, includes a single valuefor red color at each pixel location of the image. Similarly, G channelimage 520 includes a single value for green color at each pixellocation, and B channel image 530 includes a single value for blue colorat each pixel location. In some embodiments as shown in FIG. 5, for thesame image 500, different channel images may present different patternsdepending on the original color information of each component. Forexample, if component 140 is a red cup, then the R channel image 510contains the most relevant information for performing visual inspectionon component 140. On the other hand, if component 144 is a green cable,then the G channel image 520 contains the most relevant information forperforming visual inspection on the component 144.

FIG. 6A illustrates a process of preparing one or more masks (e.g.,masks 610, 612, and 614) corresponding to one or more components (e.g.,objects 140, 144, and 142) of the product 122 for visual inspection, inaccordance with some embodiments. FIG. 6B is a flow diagram illustratinga method of preparing one or more masks corresponding to one or morecomponents of a product for visual inspection, in accordance with someembodiments. The method of FIG. 6B are performed by three entitiesincluding an edge device (e.g., the camera 104-1), the server system154, and the mobile device 156. In some embodiments, a user of themobile device 156 sends (622) a command, e.g., an “initialization”command to initiate the mask preparation process, to the server 154.Upon receiving (624) the command, the server sends (626) an instruction,e.g., a “capturing image” command, to an edge device, such as the camera104-1, to capture an image. Upon receiving (628) the instruction tocapture an image, the edge device 104-1 captures (630) an image andsends the captured image to the server 154. The server receives (632)the captured image from the edge device 104-1, and further compresses(634) the captured image to a low-resolution version to save thebandwidth of network communication between the server 154 and the mobiledevice 156. The server 154 then sends (634) the compressed image to themobile device 156. The mobile device 156 displays the image to the userto let the user to circle one or more components (e.g., objects, items,parts, etc.) using respective polygons (e.g., circles, squares,rectangles, etc.). The mobile device detects (635) one or more userinputs for circling one or more components of the product out. Themobile device then records (636) the coordinates of each polygon andsends (636) it to the server 154. The server receives and uses (637) thecoordinates of each polygon corresponding to a component circled by theuser to make a masking matrix for that user selected component.

FIG. 6C is a flow diagram illustrating a method of preparing one or moremasks corresponding to one or more components of a product for visualinspection, in accordance with some embodiments. The method of FIG. 6Ccan be performed on a standalone device, e.g., a camera coupled to acontroller for performing image processing. The method starts by sending(640) the initial command for capturing an image to the controller usingthe mobile device. The controller receives (642) the initialinstruction, captures an image of the product on the assembly line, andcompresses (644) the image into a low-resolution version, and sends thecompressed image back to mobile device. The mobile device displays (646)the compressed image, and detects (646) a user input (e.g., the usercircles out each object with a respective polygon). The mobile devicerecords (648) the coordinates of each polygon input by the user, andsends it to the controller. The controller receives (650) and uses thecoordinates of each polygon corresponding to a component circled by theuser to make a masking matrix for that user selected component.

FIG. 7A illustrates a process of training one or more models for one ormore components (e.g., objects 140, 144) of a product for visualinspection, in accordance with some embodiments. In some embodiments,each model includes a plurality of sub-models (e.g., f1(x)-f2(x)) forrespective information channels (e.g., the R, G, B, and D channelsrespectively) of the image. In some embodiments, the model trainingprocess uses a blueprint of the product showing predetermined respectivepositions and orientations of the objects within the product. In someembodiments, the model training process uses a set of images captured bythe camera 104-1 of the product, and the set of images reflect thestandard positions and orientations of the objects within the product.

FIG. 7B is a flow diagram illustrating a method of training one or moremodels for one or more components of a product for visual inspection, inaccordance with some embodiments. In some embodiments, the method ofFIG. 7B is performed by the mobile device 156, the server system 154,and the edge device 104-1. In some embodiments, the user of the mobiledevice sends (720) a first request, e.g., a command “Training,” to theserver 154 using the mobile device 156. The first request is associatedwith an instruction to capture an image by a camera for training. Theserver 154 receives (722) the command from the mobile device and sends(724) a second request, e.g., a command “capturing an image” to edgedevice 104-1. The edge device 104-1 receives (726) second request, andcaptures (728) an image of the product 122 on the assembly line, andsends (728) the captured image to the server 154. The server receives(730) the captured image from the edge device 104-1, and compresses(732) the captured image to a lower-resolution image. The server 154then sends (732) the compressed image to the mobile device 156. Themobile device displays (734) the compressed image and detects (736)whether a user input received on the displayed image indicates that thecaptured image is OK. In accordance with a determination that the userinput indicates that the image is OK (e.g., the user presses the “upload‘OK’ sample” button to indicate that the sample image reflects thestandard position and orientation of the objects), the mobile devicesends a message to the server 154 and the server saves (738) this imageas a OK sample for training. In accordance with a determination that theuser indicates that the image is not OK (e.g., the user presses the“upload ‘NG’ sample” button, and enters the serial number of themissing/defective part.), e.g., missing an object or including a surfacedefect, the mobile device further receives (740) a user input regardingthe detected defects. The mobile device sends a message including theserial number of the missing/defective part to the server and the servermarks and saves (742) this image as a “Not Good” sample and the serialnumber of the missing/defective part. The system repeats the steps720-742 to gather a plurality of images 702 for training. After theserver 154 saves enough amount of training sample images, the mobiledevice 156 sends (744) a third request, e.g., a command “Train model,”to start training to the server 154. The server applies (746) the masks610, 612, and 614 to each training sample image, to obtain masked imagefor each component using the masking matrix of the correspondingcomponent. In some embodiments, the server uses (748) machinelearning/deep learning method to train the model for each componentusing the masked images for the corresponding component.

FIG. 7C is a flow diagram illustrating a method of training one or moremodels for one or more components of a product for visual inspection, inaccordance with some embodiments. The method of FIG. 7C can be performedon a standalone device, e.g., a camera coupled to a controller forperforming image processing. The mobile device sends (750) a command tocapture an image to controller. The controller controls (752) the camerato capture an image and compresses (754) the captured image into alow-resolution version and send the compressed image back to mobiledevice. The mobile device displays (756) the image to the user anddetects (758) a user's feedback regarding whether the image is OK. Ifthe user indicates that it is a “OK” sample, the mobile device sends the“OK” sign to the controller. The controller marks (760) the image as a“OK” sample. If it is a “NG” (not good) sample, the mobile devicereceives (762) the user input regarding the defects (e.g., entering aserial number of a missing/defective part). The mobile device sends the“NG” message and the serial number to the controller. The controllermarks (764) the image as “NG” sample with the serial number of theidentified defective part. Steps 750-764 are repeated to collect enough“OK” and “NG” samples. When enough samples are collected, the mobiledevice sends (766) a command “Training” to the controller to starttraining. The controller masks (768) each part of the sample by usingthe masking matrix corresponding to each part. The controller uses (770)the masked “OK” samples and “NG” samples to train the models.

FIGS. 8A-8B illustrate a process of performing product inspection usingmodels obtained from the process as discussed with reference to FIGS.7A-7C, in accordance with some embodiments. In some embodiments as shownin FIG. 8A, a test image 802 captured by the camera 104-1 is firstseparated into different sub-portions 804, 806, and 808, correspondingto respective objects 140, 144, and 142 of the product. In someembodiments, the sub-portions are selected by user inputs 801 receivedon the test image 802 displayed to the user. Alternatively, the systemcan perform image recognition to automatically identify the respectivesub-portions. In some embodiments, a plurality of information channelsfor each sub-portion (e.g., information channels 810, 812, 814, and 816)are obtained. For example, the information channels 810, 812, 814, and816 correspond to R, G, B, D channels respectively. For eachsub-portion, a corresponding sub-model obtained from the process ofFIGS. 7A-7C are used to perform the inspection. For example, thesub-model f1(s) is applied to the R channel of the sub-portion 810 toobtain a testing result 826 indicating a possibility that object 140 isdefective or missing (would use different sub-models corresponding tothe likelihood of the object 140 being defective or missingrespectively). After obtaining the testing results 826, 828, 830, and832 for each information channel of the sub-portion 810, respectiveweights w1, w2, w3, and w4 are assigned to the respective testingresults. The respective weights can be assigned by the user based onuser's experience, or automatically by the system based on previoustesting results of the same sub-portion. Then the final result 833 canbe obtained. Similarly, as shown in FIG. 8B, the information channels834, 836, 838, and 840 correspond to R, G, B, D channels of thesub-portion 144 can be obtained respectively. The respective sub-modelsfrom the process of FIGS. 7A-7C are used to perform the inspection ofthe sub-portion 144 to obtain the testing results 850, 852, 854, and856. Respective weights w5, w6, w7, and w8 are assigned to therespective testing results to calculate the final result 860.

FIG. 8C is a flow diagram illustrating a method of performing productinspection using models, in accordance with some embodiments. In someembodiments, the method of FIG. 8C is performed by the mobile device156, the server system 154, and the edge device 104-1. In someembodiments, the user using the mobile device 156 to send (862) acommand “detection” to the server 154 to capture an image. The server154 receives (863) the command and sends (864) a second request of“capturing image” to the edge device 104-1. The edge device receives(865) the command, captures (866) an image and sends (867) it to theserver. The server receives (867) the captured image and masks (868) theimage by the masking matrix of each component. The server then applies(869) each model to the masked image, by letting the masked image passthe corresponding model and calculate the result. The server sends (870)the compressed image and the detecting result to the mobile device. Themobile device displays (870) the image and the detecting result to user.The mobile device then notifies (872) the user that the system is readyfor a next request for testing, and proceeds back to step 862.

FIG. 8D is a flow diagram illustrating a method of performing productinspection using models, in accordance with some embodiments. The methodof FIG. 8D can be performed on a standalone device, e.g., a cameracoupled to a controller for performing image processing. The mobiledevice sends a user command to capture an image to the controller. theController captures (882) one image. The controller applies (884)corresponding models to the capture image, by letting the image passeach trained model and calculate the result based on each of the weight.The controller compresses (886) the image to a low-resolution version,and sends the detection result and the compressed image to the mobiledevice for user to review. The mobile device displays (888) the resultand the image. The system repeats the steps 880 to 888 for moreinspection processes.

FIG. 9A is a flow diagram illustrating a method of training one or moremodels for performing visual inspection of a product, in accordance withsome embodiments. For the training process shown in FIG. 9A, all theoperating instructions are given by the user on the client end (e.g.,the mobile device 156), and then be transformed to the server end (e.g.,the server 154). The training process includes setting communicationinformation for the server and the GUI, and then connect them together.For example, the client sends a connection request, and in response tothe connection request, the server and the client are connected. Theclient then sends a request to capture an image. The server instructsthe camera to capture an image in response to this request, anddesignates the captured image as a training sample. The server sends thecaptured image to the client for display. The client displays the imageto the user, and the user labels the location of the target component onthe image. The user may further label the anchor position (e.g., theposition of the reference marker 146). The client sends the positionlabel information to the server. The server formats the labelinformation and stores such information at the server. In someembodiments, the server captures the image and the label data for boththe standard product and the defective product, and the defectiveproduct may include different types of defects. For different types ofdefects, different labels are assigned. A plurality of training samplesare used for each type of defect.

In response to receiving a training request from the client, the serverloads the plurality of pictures with two types of labels respectively totrain a two categories classification for each classifier. That is, themodel under training can be used for identifying whether a testingsample is defective or without any defect. In some embodiments, onetraining sample is selected for position detection, and the image issegmented into the position for the defect area and the detect anchor.The server trains classification model with all the data collectedabove, and each time the system will load only one type of defectpictures, and then the system will train a 2-class classifier which canclassify whether the target area is perfect or not. In each trainingprocess, the system will affine the picture first, then crop the targetdefect area from the picture, and transform it into vector to feed intothe model. After the training process, the number of the 2-classclassifier would be as same as the types of the defects. In someembodiments, the system can also be applied with one-class classifierfor each kinds of defect. After the training process, the models forposition detection and classification are saved at the server.

FIG. 9B is a flow diagram illustrating a method of performing productinspection using models, in accordance with some embodiments. Theinspection process includes setting communication information for theserver and the GUI, and then connect them together. For example, theclient sends a connection request, and in response to the connectionrequest, the server and the client are connected. The client then sendsa request to the server to load the position detection andclassification model. In response, the server loads the detection samplepicture and classification model. The client sends a request to performinspection. The server instructs the camera to capture a test picturefor inspection. The server then finds the key points with the detectionmethod, calculates the affine matrix, and transforms the test image withthe affine matrix (e.g., to obtain a standardized image). The serveralso crops the target defect areas from the captured image, andtransforms it into input vector for the classification model. The serverclassifies the cropped image with vector with corresponding classifier.Then the server sends the result to the client for display, and the usercan review the result and provide feedbacks on the GUI of the clientdevice.

FIG. 10 is an example user interface 1000 for a user to interact withtraining models and performing inspection of a product using a mobiledevice (e.g., the mobile device 156, FIG. 1B) with a touch screen 1022,in accordance with some embodiments. In some embodiments as discussed inthe present disclosure, the user can user the mobile device to provideuser feedbacks to each training image captured by the camera 104-1, andprovide user instruction training respective models for the components(e.g., parts, objects, sub-portions, etc.). Furthermore, the user canprovide instruction for training models (e.g., 2-classifier models) foreach type of defect related to each component of the product. In someembodiments, after initiating the training process, the camera 104-1captures one or more training images (e.g., training images 702, FIG.7A). The mobile device then displays each training image on the userinterface 1000. In some embodiments, the user interface 1000 includes aplurality of icons 1002, 1004, and 1006 corresponding to the components,e.g., objects 140, 144, and 142, of the product. For example, when userselects the icon 1002, the mobile device 156 sends an instruction to theserver to initiate training of a model for the corresponding component140. The user may then provide inputs regarding whether thecorresponding component in the currently displayed training image is OKor not OK. If the user selects the OK button 1008, the user sends amessage to the server indicating that the object 140 as shown in thecurrent training image is without defect. On the other hand, if the userselects the NOT OK button 1010, the user sends a message to the serverindicating that the object 140 in the current training image isdefective.

In some embodiments during an inspection process, the mobile devicedisplays a testing image captured by the camera, and the user can selecta certain component of the product for testing. The user can furtherchoose to inspect the selected component by a certain type of defect.For example, the user may select to inspect object 140 of the product bypressing the button 1002. The user then selects a button 1016corresponding to a test of whether the selected component has a crack.As shown in FIG. 10, the system can also test for other types ofdefects, such as whether the selected component is present or missing1012, whether the selected component is flipped 1014, whether theselected component is a wrong item 1018. In some embodiments afterreceiving the result of the current test (e.g., test 3 for cracks), theUI can display a button 1020 for the user to choose to add a new testfor a new type of defect, such as “rotated.”

FIG. 11A is a flow diagram illustrating a method 1100 of performing anautomated object inspection on an assembly line, in accordance with someembodiments. In some embodiments, the steps of the method 1100 areperformed by a server system (e.g., server system 152, FIG. 1B; serversystem 200, FIG. 2) that serves one or more assembly lines or qualityassurance stations. In some other embodiments, the steps of method 1100are performed by a mobile device (e.g., mobile device 156, FIG. 1B) thatis carried by mobile quality assurance personnel. In some otherembodiments, the steps of method 1100 are performed by one or more edgedevices (e.g., edge devices 104-1, 104-2, . . . 104-n, FIGS. 1A-1B)located at a product assembly line or quality assurance station. In someembodiments, the steps of the method 1100 may be performed by anycombination of one or more edge devices, the server system, and themobile device.

In some embodiments, the edge device 104 includes a camera (e.g., acolor camera, a black/white camera, an infrared camera) for capturingimages of assembled products or accessories. In some embodiments, theedge devices include various other types of sensors (e.g., weightsensors, touch-sensors, vibration sensors, sound sensors, hapticsensors, temperature sensors, light sensors, movement sensors, etc.) forcollecting information about the state of the assembled products andassembly lines. In some embodiments, the edge devices are positioned atfixed locations around the assembly lines or inspection stations. Insome embodiments, the edge devices are actuated and positioned by one ormore robotic arms around the assembly lines or inspection stations.Method 1100 of FIG. 11A corresponds to instructions stored in a computermemory or computer readable storage medium (e.g., memory 206 of theserver system 200, FIG. 2).

In performing the method 1100, the server system obtains (1102) aplurality of test images (e.g., including a test image as shown in FIG.4A) that are captured by an edge device (e.g., a camera 104, FIG. 1A)located over an assembly line (e.g., the assembly line 100, FIG. 1A) ora quality assurance station. In some embodiments, each test imagecaptures a respective one of a plurality of composite objects. In someembodiments, the respective composite object (e.g., the product 122,FIG. 1A) corresponds to a multi-component product. In some embodiments,the respective composite object corresponds to an accessory packincluding multiple objects (e.g., the accessory pack 122 includingmultiple objects 140, 142, and 144, FIG. 1A) that have been assembled inaccordance with a predefined common layout. In some embodiments, thecommon layout is a blueprint of the multi-component product or theaccessory pack. Alternatively, the common layout is an image captured bythe camera located over the assembly line of a standard example of themulti-component product or the accessory pack placed at a standardlocation (e.g., upright and squarely aligned in the center of the table)on the assembly line or quality assurance station for the plurality ofcomposite objects.

The predefined common layout specifies (1104) respective positions formultiple components of a respective composite object, and a predefinedlocation and orientation for a reference marker (e.g., reference marker146, FIG. 1A) in the respective composite object. In some embodiments, acomposite object is a product including multiple parts that areconnected to one another in accordance with a predefined design. Forexample, a composite object includes an electric fan including a headcoupled to a platform. In another example, a composite object includes arefrigerator with crisper boxes and wire racks inside. In someembodiments, the predefined common layout specifies respective positionsfor various accessories (e.g., electric cables, battery pack,instruction manual, a tool kit, etc.) that are packed in variouscompartments of the accessory box (e.g., the accessory pack 122, FIG.1A). In some embodiments, the reference marker is a specially designedprinted mark, hole, or protrusion, that is designed to have a fixedlocation and orientation on the product or the accessory box, such thatthe orientation and lateral position of the product or the accessory boxcan be deduced accurately once the location and orientation of thereference marker is determined. In some embodiments, for each testimage, the orientation and lateral position of the reference marker canbe determined based on image processing and object recognitiontechniques (e.g., discussed with reference to FIGS. 4A-4J). In someembodiments, the reference marker has an asymmetrical shape such thatthe reference marker shows distinct and unique patterns in variousorientations.

Each test image captures (1106) respective portions of a correspondingcomposite object that contains the two or more components and thereference marker, and captures at least a portion of a surroundingenvironment of the corresponding composite object. For example, eachtime when a newly assembled product or accessory pack is placed on atable along the assembly line or the quality assurance station, thecamera (e.g., the camera 104-1, FIG. 1A) placed at a fixed location orcontrolled by a robotic arm captures an image of the product oraccessory pack from above. In an actual production scenario, thelocation and orientation of the product or the accessory pack are notalways exactly the same. For example, the products or the accessorypacks may be dropped and/or pushed into the camera view after they areassembled, and they may be of slightly different lateral positions androtational angles on the table, even if they are all laid flat on thetable. As discussed with reference to FIGS. 3B-3E, in some embodiments,the purposes of the inspection via the images include inspection fordefects (e.g., visual defects or surface defects) including, but notlimited to, scratches, abrasion, corrosion, debris, cracks, blemishes,broken off pieces. In some embodiments, the purposes of the inspectionvia the images include inspection for missing parts, such as missingitems in certain holes of the accessory box, missing a screw in a screwhole. In some embodiments, the purposes of the inspection via the imagesinclude inspection for inaccurate positioning of the parts, for example,when wire rack of the refrigerator is not properly inserted into thesliding track, or when electric cable and battery pack are swapped intheir respective compartments in the accessory box.

The server system further processes each of the plurality of test imagesas discussed in the following processes. In some embodiments, theprocessing of the images can be performed serially as each product oreach accessory pack is placed on the inspection table along the assemblyline and a picture of the product or the accessory pack is taken. Insome embodiments, the images from multiple product lines may beprocessed in parallel.

For each of the plurality of test images, the server systemquantitatively determines (1110) a respective transformation from apredefined contour of the reference marker in the predefined commonlayout to a respective anchor contour corresponding to the referencemarker as captured in the test image. In some embodiments, thepredefined contour of the reference marker is specified in a blueprintof the product or the accessory box design. In other embodiments, thepredefined contour of the reference marker is obtained based on adescription of the reference marker that is extracted based on machinelearning from a plurality of training images each of which includes thereference marker on sample composite products or sample accessory packs.In some embodiments, the respective anchor contour is extracted andrecognized from the captured test image based on image processingtechniques described herein.

In some embodiments, quantitatively determining a respectivetransformation from the predefined contour of the reference marker inthe predefined common layout to the respective anchor contourcorresponding to the reference marker as captured in a test imageincludes: quantitatively determining a translational offset and arotational angle between the reference marker in the predefined commonlayout and the reference marker in the captured test image. In someembodiments, the transformation is described as a mathematicaltransformation (e.g., an affine matrix) within a fixed externalcoordinate system (e.g., a 2-dimensional coordinate system with anorigin at the center of the reference mark in the predefined commonlayout, e.g., as discussed with reference to FIGS. 4C-4D.

Continuing to step 1112, for each of the plurality of test images, theserver system applies (1112) an inverse of the respective transformationto at least a portion of the test image to obtain a respectiveregularized version of the corresponding portion of the test image, suchthat the reference markers captured in the respective regularizedversions of the plurality of test images share an image-independentlocation and orientation. For example, after reversing the respectiverotation and translation experienced by each composite object relativeto the standard location and orientation of the reference marker in thepredefined common layout in the test image, e.g., by applying theinverse of the respective transformation that has been obtained in theprevious step on the test image, the reference marker becomes the anchorby which the test image is shifted and rotated such that all thecomponents of the composite object are located at their respectivelocations according to the predefined common layout, and can be easilyextracted for recognition independent of the other components.

In some embodiments, quantitatively determining a respectivetransformation from the predefined contour of the reference marker inthe predefined common layout to a respective anchor contourcorresponding to the reference marker as captured in the test imagecomprises: (1) extracting a plurality of contours from the test image(e.g., by binarizing the test image, and identifying pixels alongboundaries between pixel value transitions); (2) calculating acorresponding total area of each of the plurality of contours extractedfrom the test image (e.g., by first connecting adjacent pixels that aresituated at pixel value transition boundaries to obtain an enclosed areafor calculating the total area); and (3) identifying the respectiveanchor contour from among the plurality of contours extracted from thetest image in accordance with a determination that a difference betweena total area of the reference marker in the predefined common layout andthe corresponding total area of the anchor contour meets area-differencecriteria. For example, the area-difference criteria require that adifference in total area between two contours in comparison is below athreshold value (e.g., ±10%, ±5%, ±1%, ±0.5%, ±0.2%, etc.) in order forthe area-difference criteria to be met.

In some embodiments, when the composite object is a complex object withcomplex parts, a lot of contours will be detected in the test imageusing conventional simple contour detection methods. The contours thatare detected within the test image may correspond to internal structuresof the components, the outlines of the components, or shadows, as wellas the reference marker in the composite object. Using the total area tofilter out contours that correspond to internal structures and shadows,which typically make small contours or large contours that do not matchthe size of the reference marker, is a quick and simply way to identifyjust a few good candidate contours for the reference marker from a largenumber of contours detected in the test image. Furthermore, total areais not affected by the translation and rotation of the composite objectwhen the test image is captured, and serves as a good first filter formore detailed image analysis.

In some embodiments, the reference marker is designed to have a shapeand size that are distinct from most common shapes and sizes of thecontours that may result from other components and structure of thecomposite object. For example, the reference marker may be designedafter a large number of training images of sample composite objects areanalyzed and distribution of the total areas of the contours detected inthe training images statistically measured. In some embodiments, thereference marker is optionally designed to have a total size that is atleast 10% different from the sizes of over 90% of the contours detectedin the training images. For example, the threshold difference isoptionally set at a 10% difference of the size of the reference marker.In some embodiments, the size of the reference marker is adjustable fordifferent composite object by printing or pressing a mark on the surfaceof the product or accessory box. In some embodiments, the design of thereference mark is based on the available space on the product oraccessory box, a rectangular shape or other polygonal shapes withnon-uniform edge lengths may be preferable because it is easy to changethe size and easy to identify a rotation of the shape.

In some embodiments, identifying the respective anchor contour fromamong the plurality of contours extracted from the test image furthercomprises: identifying, from among the plurality of contours extractedfrom the test image, two or more candidate anchor contours that meet thearea-difference criteria. For example, from among all the contoursextracted from the test image, identify the subset of contours that havetotal areas within 10% difference from the total area of the referencemarker in the predefined common layout, and the subset of contours areused as candidate contours for subsequent shape evaluation.

In some embodiments, identifying the respective anchor contour fromamong the plurality of contours extracted from the test image furthercomprises: selecting the anchor contour from the two or more candidatecontours based on relative closeness between the respective shapefeature of each of the two or more candidate contours and a shapefeature of the reference mark in the predefined common layout. There aremore than one way to match the shape features of the candidate contoursto the shape feature of the reference mark. In some embodiments, thecenter coordinates of each contour are determined, and the shape featureis calculated based on the number of local maximum distances from thecenter to each point along the candidate contour (e.g., the numberrepresents the number of apexes of the shape). In some embodiments, thecandidate contour with a shape that has the same number of apexes as thereference mark is selected as the anchor contour. In some embodiments,the contours are smoothed first (e.g., with a running average) beforethe distance between each point along the contour and the center of thecontour is calculated. In some embodiments, the closeness of the shapeis determined based on a shape model. For example, the shape feature ofthe reference marker is learned by the computer using training imagescontaining only the reference marker (e.g., the image is a cropped imagecontaining only the reference mark), and the portions of test imagescontaining the candidate contours are cropped out and used as input forshape recognition. The candidate contour that is recognized as thereference marker by the computer is selected as the anchor contour. Insome embodiments, shape distances are computed between each candidatecontour and the reference marker in the predefined common layout, andthe candidate contour that has the smallest shape distance from thereference marker is selected as the anchor contour. Regardless of thetechnique that is used to identify the anchor contour from the candidatecontour, the technique must be designed such that the identification canoccur even when the reference markers in the test images are shiftedand/or rotated relative to the position and orientation as indicated inthe predefined common layout.

In some embodiments as discussed with reference to FIGS. 4D-4F and 4J,quantitatively determining a respective transformation from a predefinedcontour of the reference marker in the predefined common layout to arespective anchor contour corresponding to the reference marker ascaptured in the test image includes: (1) obtaining center coordinates ofthe respective anchor contour in the test image (e.g., after therespective anchor contour corresponding to the reference marker ascaptured in the test image is identified; (2) calculating a translationoffset (e.g., the translation part of the transformation) between centercoordinates of the respective anchor contour in the test image andcenter coordinates of the reference marker in the predefined commonlayout; (3) generating a first histogram Di(Theta′) of distances betweenthe center coordinates of the respective anchor contour and each of asequence of n equally spaced points around the respective anchor contour(e.g., adjacent points are 360/n degrees apart or 360/m degrees apartwith m>n and n points sequentially located to each other) (e.g., FIG.4E); (4) generating a second histogram Di (Theta) of distances betweenthe center coordinates of the reference marker and each of a sequence ofn equally spaced points around a contour of the reference marker asspecified in the predefined common layout (e.g., FIG. 4F); (5)determining an angular shift (AO) on the first histogram that will causethe histogram to achieve alignment with the second histogram (e.g., whenthe distance between the first histogram and the second histogram in acontinuous 360-degree range reaches a minimum value with that angularshift applied to the first histogram); and (6) using the translationoffset and the angular shift as the respective transformation from thepredefined contour of the reference marker in the predefined commonlayout to the respective anchor contour corresponding to the referencemarker as captured in the test image.

The server system further performs (1114) independent automatedinspection on each of two or more sub-portions of the respectiveregularized versions of the test images. In some embodiments, eachsub-portion of the respective regularized versions of the test imagescorresponds to a respective one of the two or more components of therespective composite object or of two or more objects of the accessorypack captured in the test image. In some embodiments, each sub-portionof the test image is extracted using a mask (e.g., mask 602, 604, and606, FIG. 6A) corresponding to the sub-portion. The mask opening hascoordinates that are offset by predefined distances (e.g., in the x andy directions) from the reference marker (e.g., the center of thereference mark) in the predefined common layout. The extracted portionof the test image are processed using an independent image analysismodel that are specially trained for recognizing the component or itspresence/absence, or examining its state or defects, etc., depending onthe requirements of the production line. Once the independent imageanalysis for each of the two or more sub-portions of the respectiveregularized versions of the test image has been performed, the resultsof the image analysis are integrated to generate a report. For example,if a particular accessory is determined to be missing from one of thecompartment of the accessory box based on the image analysis of theportion of the test image that corresponds to the compartment designedfor holding that particular accessory, an error report is generatedindicating the type of accessory that is missing from the accessory boxfor the accessory pack corresponding to the currently analyzed testimage. Once the report is generated, the accessory box can be picked upby a robot arm or a worker for remediation, e.g., adding the requiredaccessory. If everything seems in order with respect to the imageanalysis on all the sub-portions of the test image, a report isgenerated indicating that the composite object corresponding to thecurrent test image has passed inspection.

In some embodiments, quantitatively determining a respectivetransformation from the predefined contour of the reference marker inthe predefined common layout to a respective anchor contourcorresponding to the reference marker as captured in the test imagecomprises: after the respective anchor contour corresponding to thereference marker as captured in the test image is identified, obtainingan ordered sequence (e.g., a clockwise or counterclockwise sequence) ofkey coordinates (e.g., the four corners) for a minimum boundingrectangle (e.g., a bounding box or envelope) of the respective anchorcontour in the test image (e.g., FIG. 4C). In some embodiments,obtaining the minimum bounding rectangle can be achieved using any knownmethods for determining the maximum extends of a two-dimensional object(e.g., the anchor contour). In some embodiments, the server system alsoobtains an ordered sequence of key coordinates for a minimum boundingrectangle of the reference mark in the predefined common layout (e.g.,FIG. 4C). In some embodiments, based on the ordered sequences of keypoints for the minimum bounding rectangles of the reference mark in thepredefined common layout and the anchor contour in the test image, theserver system determines a respective affine transformation (e.g., theaffine transformation is the respective transformation from a predefinedcontour of the reference mark in the predefined common layout to arespective anchor contour corresponding to the reference mark ascaptured in the test image) from the minimum bounding rectangle of thereference mark in the predefined common layout to the minimum boundingrectangle of the respective anchor contour.

In some embodiments, applying an inverse of the respectivetransformation to at least a portion of the test image to obtain arespective regularized version of the test image, such that thereference markers captured in the respective regularized versions of theplurality of test images share an image-independent location andorientation, further comprises: applying an inverse of the affinetransformation (as determined above) to at least a portion of the testimage. For example, the inverse of the affine transformation is appliedto a minimum bounding rectangle that is defined based on the location ofthe minimum bounding rectangle of the anchor contour in the test imagethat will include all the components of the composite object and thereference mark based on the locations and dimensions set forth in thepredefined common layout. In some embodiments, the portion of the testimage includes the reference mark and the two or more components of thecomposite object.

In some embodiments, performing independent automated inspection on eachof two or more sub-portions of the respective regularized versions ofthe test images includes: for the respective regularized version of eachof the plurality of test images, (1) extracting a first sub-portion ofthe respective regularized version of the test image at a locationcorresponding to a first component of the two or more components inaccordance with a location of the first component as specified in thepredefined common layout; (2) extracting a second sub-portion of therespective regularized version of the test image at a locationcorresponding to a second component of the two or more components inaccordance with a location of the second component as specified in thepredefined common layout; (3) performing object inspection on the firstsub-portion of the respective regularized version of the test imagebased on a first model trained on training images containing the firstcomponent and not containing the second component; and (4) performingobject inspection on the second sub-portion of the respectiveregularized version of the test image based on a second model trained ontraining images containing the second component and not containing thefirst component. In some embodiments, the extraction of the first orsecond sub-portion of the respective regularized version of the testimage includes using a respective mask with a window (e.g., a minimumenclosing rectangle for the first component or for the second componentrespectively, FIGS. 6A-6C) at a location that corresponds to therespective location of the first component or the second component inaccordance with the location of the first component or the secondcomponent on the predefined common layout. Alternatively, a copy of theregularized version is cropped to leave only the minimum enclosingrectangle for the first component or the second component in the image.More detailed discussion regarding automated inspection are discussedwith reference to the method 1150 of FIG. 11B below.

FIG. 11B is a flow diagram illustrating a method 1150 of performing anautomated object inspection on an assembly line, in accordance with someembodiments. In some embodiments, the steps of the method 1150 areperformed by a server system (e.g., server system 152, FIG. 1B; serversystem 200, FIG. 2) that serves one or more assembly lines or qualityassurance stations. In some other embodiments, the steps of method 1150are performed by a mobile device (e.g., mobile device 156, FIG. 1B) thatis carried by mobile quality assurance personnel. In some otherembodiments, the steps of method 1150 are performed by one or more edgedevices (e.g., edge devices 104-1, 104-2, . . . 104-n, FIGS. 1A-1B)located at a product assembly line or quality assurance station. In someembodiments, the steps of the method 1150 may be performed by anycombination of one or more edge devices, the server system, and themobile device.

In some embodiments, the edge device 104 includes a camera (e.g., acolor camera, a black/white camera, an infrared camera) for capturingimages of assembled products or accessories. In some embodiments, theedge devices include various other types of sensors (e.g., weightsensors, touch-sensors, vibration sensors, sound sensors, hapticsensors, temperature sensors, light sensors, movement sensors, etc.) forcollecting information about the state of the assembled products andassembly lines. In some embodiments, the edge devices are positioned atfixed locations around the assembly lines or inspection stations. Insome embodiments, the edge devices are actuated and positioned by one ormore robotic arms around the assembly lines or inspection stations.Method 1150 of FIG. 11B corresponds to instructions stored in a computermemory or computer readable storage medium (e.g., memory 206 of theserver system 200, FIG. 2).

In performing the method 1150, the server system obtains (1152) aplurality of test images (e.g., including a test image 802, FIG. 8A). Insome embodiments, the test images are captured by one or more edgedevices 104 (e.g., a camera 104-1, FIG. 1A) located over an assemblyline (e.g., the assembly line 100, FIG. 1A) or a quality assurancestation. In some embodiments, each test image captures a respective oneof a plurality of composite objects. In some embodiments, the respectivecomposite object (e.g., the product 122, FIG. 1A) corresponds to amulti-component product. In some embodiments, the respective compositeobject corresponds to an accessory pack including multiple objects(e.g., the accessory pack 122 including multiple objects 140, 142, and144, FIG. 1A) that have been assembled in accordance with a predefinedcommon layout. In some embodiments, the common layout is a blueprint ofthe multi-component product or the accessory pack. Alternatively, thecommon layout is an image captured by the camera located over theassembly line of a standard example of the multi-component product orthe accessory pack placed at a standard location (e.g., upright andsquarely aligned in the center of the table) on the assembly line orquality assurance station for the plurality of composite objects.

In some embodiments, the predefined common layout specifies respectivepositions for multiple components of a respective composite object, anda predefined location and orientation for a reference marker (e.g.,reference marker 146, FIG. 1A) in the respective composite object. Insome embodiments, user inputs are used to identify the differentobjects. In some embodiments, a reference marker is not necessarilyincluded in the layout. In some embodiments, the composite object is aproduct including multiple parts that are connected to one another inaccordance with a predefined design. For example, a composite objectincludes an electric fan including a head coupled to a platform. Inanother example, the composite object includes a refrigerator withcrisper boxes and wire racks inside. In some embodiments, the predefinedcommon layout specifies respective positions for various accessories(e.g., electric cables, battery pack, instruction manual, a tool kit,etc.) that are packed in various compartments of the accessory box(e.g., the accessory pack 122, FIG. 1A).

In some embodiments, each test image captures respective portions of acorresponding composite object that contains the two or more componentsand the reference marker, and captures at least a portion of asurrounding environment of the corresponding composite object. Forexample, each time when a newly assembled product or accessory pack isplaced on a table along the assembly line or the quality assurancestation, the camera (e.g., the camera 104-1, FIG. 1A) placed at a fixedlocation or controlled by a robotic arm captures an image of the productor accessory pack from above (e.g., as shown in FIG. 1A). In an actualproduction scenario, the location and orientation of the product oraccessory pack are not always exactly the same. For example, theproducts or accessory packs may be dropped and/or pushed into the cameraview after they are assembled, and they are all of slightly differentlateral positions and rotational angles on the table, even if they areall laid flat on the table. As discussed with reference to FIGS. 3B-3E,in some embodiments, the purposes of the inspection via the imagesoptionally includes inspection for defects (e.g., visual defects orsurface defects) including, but not limited to, scratches, abrasion,corrosion, debris, cracks, blemishes, broken off pieces. In someembodiments, the purposes of the inspection via the images includeinspection for missing parts, such as missing items in certain holes ofthe accessory box, missing a screw in a screw hole. In some embodiments,the purposes of the inspection via the images include inspection forinaccurate positioning of the parts, for example, when wire rack of therefrigerator is not properly inserted into the sliding track, or whenelectric cable and battery pack are swapped in their respectivecompartments in the accessory box.

The server system further processes each of the plurality of test imagesas discussed in the following processes. In some embodiments, theprocessing of the images can be performed serially as each product oreach accessory pack is placed on the inspection table along the assemblyline and a picture of the product or the accessory pack is taken. Insome embodiments, the images from multiple product lines may beprocessed in parallel.

For each of the plurality of test images, the server system performs(1154) independent object inspection on each of two or more sub-portionsof the test image (or the respective regularized versions thereof). Insome embodiments, each sub-portion of the test image corresponds to arespective one of the two or more components of the respective compositeobject captured in the test image.

In performing the independent object inspection of the test image, theserver system segments (1156) the test image (e.g., the test image 802,FIG. 8A) into at least a first sub-portion (e.g., sub-portion 804, FIG.8A) of the test image and a second sub-portion (e.g., sub-portion 806,FIG. 8A) of the test image. In some embodiments, the first sub-portionof the test image contains a first component (e.g., object 140, FIG. 8A)and not containing a second component (e.g., object 144, FIG. 8A) of thetwo or more components (e.g., objects 140, 142, and 144, FIG. 8A). Insome embodiments, the second sub-portion of the test image containingthe second component (e.g., object 144, FIG. 8A) and not containing thefirst component (e.g., object 140, FIG. 8A) of the two or morecomponents

The server system performs (1158) object inspection on the firstsub-portion (e.g., sub-portion 804, FIG. 8A) of the test image using afirst subset of information channels (e.g., information channels 810,812, 814, and 816, FIG. 8A) of the test image and a first model trainedon a first set of training images (e.g., training images 702, FIG. 7A)containing the first component (e.g., object 140) and not containing thesecond component (e.g., object 144).

The server system performs (1160) object inspection on the secondsub-portion (e.g., sub-portion 806, FIG. 8B) of the test image using asecond subset of information channels (e.g., information channels 834,836, 838, and 840, FIG. 8B) of the test image that are distinct from thefirst subset of information channels, and a second model trained on asecond set of training images containing the second component (e.g.,object 144) and not containing the first component (e.g., object 140).

In some embodiments, segmenting the test image into at least a firstsub-portion of the test image and a second sub-portion of the test imageincludes (1) detecting user inputs (e.g., user input 604, FIG. 6A)respectively directed to a first location (e.g., a circle 606, FIG. 6A)in the test image (e.g., the test image 602, FIG. 6A) and a secondlocation (e.g., a square 608, FIG. 6A) in the test image; (2) creating afirst mask (e.g., mask 610, FIG. 6A) for the test image to extract thefirst sub-portion (e.g., sub-portion 804, FIG. 8A) of the test image inaccordance with the first location; (3) creating a second mask (e.g.,mask 612, FIG. 6A) for the test image to extract the second sub-portion(e.g., sub-portion 806, FIG. 8B) of the test image in accordance withthe second location; and (4) extracting the first sub-portion of thetest image and the second sub-portion of the test image using the firstmask and the second mask, respectively (e.g., FIG. 8A). In someembodiments, the server system also detects a contour of the firstcomponent (e.g., object 140) around the first location in the testimage, and a contour of the second component (e.g., object 144) aroundthe second location in the test image. In some embodiments, the serversystem performs an initialization process (e.g., FIGS. 6A-6C) to createmasks based on coordinates for respective sub-portions before thetraining process (e.g., FIGS. 7A-7C) and the testing process (e.g.,FIGS. 8A-8D).

In some embodiments, prior to or while performing segmenting the testimage into at least a first sub-portion of the test image and a secondsub-portion of the test image, the server system applies a respectivetransformation to the test image to align a reference mark captured inthe test image and a corresponding reference mark specified in apredefined common layout of the plurality of composite objects asdiscussed with reference to FIGS. 4A-4J and 11A. For example, in someembodiments, the computer quantitatively determines an affinetransformation from a predefined contour of the reference marker in thepredefined common layout to a respective anchor contour corresponding tothe reference marker as captured in the test image. Then, the computerapplies an inverse of the affine transformation to at least a portion ofthe test image to obtain a respective regularized version of the testimage, such that the reference markers captured in the respectiveregularized versions of the plurality of test images share animage-independent location and orientation. By applying the inverse ofthe affine transformation that has been obtained in the previousprocesses (e.g., FIGS. 4A-4J and 11A) on the test image, the referencemark becomes the anchor by which the test image is shifted and rotatedsuch that all the components of the composite object are located attheir respective locations according to the predefined common layout,and can be easily extracted for recognition independent of the othercomponents

In some embodiments, after the respective transformation is applied tothe test image, the server system extracts (e.g., using a mask (such asmask 610) or mathematically specifying a minimum bounding box) the firstsub-portion (e.g., sub-portion 804, FIG. 8A) of the test image inaccordance with a location of the first component specified in thepredefined common layout of the plurality of composite objects. Theserver also extracts (e.g., using a mask (such as mask 612) ormathematically specifying a minimum bounding box) the second sub-portion(e.g., sub-portion 806, FIG. 8A) of the test image in accordance with alocation of the second component specified in the predefined commonlayout of the plurality of composite objects.

In some embodiments, performing object inspection on the firstsub-portion of the test image using a first subset of informationchannels of the test image and a first model trained on a first set oftraining images containing the first component and not containing thesecond component, includes separating test image into respectivesingle-channel images corresponding to a plurality of informationchannels of the test image. Correspondingly, each of the sub-portions ofthe test image is also separated into a corresponding set ofsingle-channel sub-portions of the test image for the plurality ofinformation channels. For each information channel of the first subsetof information channels, the server system obtains a first sub-portionof the respective single-channel image corresponding to said eachinformation channel. In some embodiments, the first sub-portion of therespective single-channel image corresponds to the first sub-portion ofthe test image containing the first component. In some embodiments, theserver system performs object inspection on the first sub-portion of therespective single-channel image of the test image corresponding to saideach information channel, using a respective single-channel sub-model(e.g., sub-model f1(x), f2(x), f3(x), and f4(x), FIGS. 7A and 8A) of thefirst model that is trained (e.g., FIGS. 7A-7C) on single-channeltraining images of the first component captured with said eachinformation channel.

In some other embodiments, the server system first segments (e.g., FIG.8A and step 1156 of FIG. 11B) the test image into a plurality ofsub-portions, including the first sub-portion 804 and the secondsub-portion 806. Then the server system separates each sub-portion intorespective single-channel sub-portions (e.g., single-channel sub-portion810, 812, 814, and 816, FIG. 8A) corresponding to a plurality ofinformation channels of each sub-portion of the test image. Then theserver system performs object inspection on the respectivesingle-channel sub-portions corresponding to said each informationchannel, using a respective single-channel sub-model (e.g., sub-modelf1(x), f2(x), f3(x), and f4(x), FIGS. 7A and 8A) of the first model thatis trained (e.g., FIGS. 7A-7C) on single-channel training images of thefirst component captured with said each information channel.

For example, in an example scenario where the test image includes fourinformation channels (RGBD), representing three color channels (RGB) andone depth channel (D), the test image is first separated into foursingle-channel images each corresponding to a respective one of the fourchannels. The single-channel image of each channel includes a singlevalue for that channel at each pixel location of the test image. Foreach information channel of the first subset of information channels(e.g., the R channel and the D channel, respectively), object inspectionis performed on the first sub-portion of the correspondingsingle-channel image of the test image (e.g., the single-channel imagesfor the R channel and D channel, respectively), using a single-channelsub-model of the first model (e.g., the sub-model for the R channel andthe sub-model for the D channel, respectively) that is trained onsingle-channel training images of the first component for that channel(e.g., R channel images of the first component, and D channel images ofthe first component, respectively).} {In some embodiments, performingobject inspection on the second sub-portion of the test image using asecond subset of information channels of the test image and a secondmodel trained on a second set of training images containing the secondcomponent and not containing the first component, includes: afterseparating the test image into respective single-channel imagescorresponding to the plurality of information channels of the testimage, for each information channel of the second subset of informationchannels that are distinct from the first subset of informationchannels: obtaining the second sub-portion of the respectivesingle-channel image corresponding to said each information channel,wherein the second sub-portion of the respective single-channel imagecorresponds to the second sub-portion of the test image that includesthe second component; and performing object inspection on the secondsub-portion of the respective single-channel image of the test imagecorresponding to said each information channel, using a respectivesingle-channel sub-model of the second model that is trained onsingle-channel training images of the second component captured withsaid each information channel. Continue with the example of the testimage with the four channels, for each information channel of the secondsubset of information channels (e.g., the G channel and the D channel,respectively), object inspection is performed on the second sub-portionof the corresponding single-channel image of the test image (e.g., thesingle-channel images for the G channel and D channel, respectively),using a single-channel sub-model of the second model (e.g., thesub-model for the G channel and the sub-model for the D channel,respectively) that is trained on single-channel training images of thesecond component for that channel (e.g., G channel images of the firstcomponent, and D channel images of the first component, respectively).

In some embodiments, the composite object is an accessory box 122containing a red cup (e.g., object 140) in a first round compartment anda green cable (e.g., object 144) in a second square compartment. Thetest image capturing the composite box is preprocessed to align thelocations of the compartments in the test image with a predefined layout(e.g., based on location of a reference mark on the box as discussedwith reference to FIGS. 4A-4J and 11A). The first sub-portion 804 of thetest image corresponds to the first compartment 140 and the secondsub-portion 806 of the test image corresponds to the second compartment144. Because the component of interest in the first compartment is a redcup, the channels that are most relevant to the inspection (e.g.,presence or absence of the component, whether the cup is properlyinserted into the compartment) are the R channel and D channel, whilethe G and B channels contain little relevant information in theinspection of the first sub-portion of the test image. Similarly, thecomponent of interest in the second compartment is a green cable, thechannels that are most relevant to the inspection are the G channel andD channel, while the R channel and B channel contain little relevantinformation in the inspection of the second sub-portion of the testimage.

In some embodiments, when processing the first sub-portion of the testimage, the first sub-portion of the single-channel image for the Rchannel is processed by a single-channel sub-model (e.g., f1(x)) that istrained on training images of the red cup in the first compartmentwithout the interference of image data on the second compartment or thegreen cable, or the interference of data from the GBD channels. Inaddition, the first sub-portion of the single-channel image for the Dchannel is processed by a single-channel sub-model (e.g., f4(x)) that istrained on training images of the red cup in the first compartmentwithout the interference of image data on the second compartment or thegreen cable, or the interference of data from the RGB channels. Whenprocessing the second sub-portion of the test image, the secondsub-portion of the single-channel image for the G channel is processedby a single-channel sub-model (e.g., f6(x)) that is trained on trainingimages of the green cable in the second compartment without theinterference of image data on the first compartment or the red cup, orthe interference of data from the RBD channels. In addition, the secondsub-portion of the single-channel image for the D channel is processedby a single-channel sub-model (e.g., f8(x)) that is trained on trainingimages of the green cable in the second compartment without theinterference of image data on the first compartment or the red cup, orthe interference of data from the RGB channels. By isolating theindividual components, and the individual information channels, themodels are simpler, faster, and more sensitive (e.g., due to lessinterference of irrelevant information from other objects or channels),producing higher inspection accuracy.

In some embodiments, performing object inspection on the firstsub-portion of the test image using a first subset of informationchannels of the test image and a first model trained on a first set oftraining images containing the first component and not containing thesecond component, includes generating an inspection result indicating atleast one of presence or position of the first component in thecomposite object by combining inspection results from respectiveinformation channels of the first subset of information channels withcorresponding weights for the respective information channels.

In some embodiments, the weights are assigned to the different channelsfor the inspection of different components of the composite object basedon relative recognition accuracies for each of the different channelsbased on training performed on the different channel for differentcomponents of the composite object. In some embodiments, the weights areassigned by a user who has visually inspected the components of thecomposite object and knows which channels are most effective ininspecting each component of the composite object. In some embodiments,the weights for the respective channels for the first component may bedifferent from the weights for the respective channels assigned to thesecond component.

For example, for the first component, the result from the R channelindicates that there is a 50% chance that the first component is presentin the first compartment in the test image, and result from the Dchannel indicates that the first compartment is 40% filled, therespective weights assigned to the R channel and D channel for the firstcomponent are 0.7 and 0.3 respectively, thus the combined resultindicates that the first compartment is not properly filled (e.g.,0.5*0.7+0.4*0.3=0.47<threshold score). However, if the result from the Rchannel indicates that there is a 80% chance that the first component ispresent in the first compartment in the test image, and result from theD channel indicates that the first compartment is less than 40% filled,and the combined result indicates that the first compartment is properlyfilled (e.g., 0.8*0.7+0.4*0.3=0.70>threshold score).

For the second component, the weight may be different (e.g., D channelis less import) for the combination of results because thecharacteristics of the components are different. For example, therespective weights assigned to the G channel and D channel for the firstcomponent are 0.9 and 0.1 respectively. The result from the G channelindicates that there is a 50% chance that the second component ispresent in the second compartment in the test image, and result from theD channel indicates that the second compartment is 40% filled, and thecombined result indicates that the second compartment is not properlyfilled (e.g., 0.5*0.9+0.4*0.1=0.49<threshold score). However, if theresult from the G channel indicates that there is a 80% chance that thesecond component is present in the second compartment in the test image,and result from the D channel indicates that the first compartment isless than 40% filled, and the combined result indicates that the firstcompartment is properly filled (e.g., 0.8*0.9+0.4*0.1=0.76>thresholdscore).

In the above examples, the components are pure colored, and the channelinformation is dominated by one color channel. In a more common usagescenario, each of the channels would have some information content thatare relevant to the inspection result, and the channels are selectedbased on relative recognition accuracies for each of the differentchannels based on training performed on the different channel fordifferent components of the composite object.

In some embodiments, the computing device is a server that is incommunication with an edge device or a peripheral device (e.g., one ormore cameras configured to capture images of the composite objects onthe assembly line) and that is in communication with a user device(e.g., a mobile device configured to receive user inputs and displayimages to the user). In some embodiments, the server system receives auser instruction for training the first model. In accordance withreceiving the user instruction for training the first model, the serversystem trains images using the edge device. In some embodiments, theserver system further receives a user input marking respective firstportions of the training images and corresponding labels for the firstportions of the training images. In some embodiments, the correspondinglabels include a first classifier and a second classifier for the firstcomponent captured in the training images. In some embodiments, thefirst classifier and the second classifier are Pass (P), No Pass (NP)labels for a first parameter of inspection for the first component(e.g., the first parameter may be object presence, object position,object non-defective, etc.). In some embodiments, the server systemfurther generates the first model based on the respective first portionsof the training images and the corresponding labels provided in the userinputs. In some embodiments, the training of the first model isperformed based on first portions of single-channel images of each ofmultiple information channels of the training images. The trainingresults in multiple single-channel sub-models of the first model, withdifferent inspection accuracies. The relative accuracies of thesub-models are used to assign different weights to the differentinformation channels and selecting the most relevant informationchannels for subsequent inspection of test images.

In some embodiments, in accordance with a determination that a modelaccuracy of the first model meets a first threshold accuracy, the serversystem presents a notification on the user device indicating thattraining of the first model is complete. For example, once the userknows that the training of the first model is complete, the user canfocus her time and effort on training other types of inspection issues,or other components of the composite object for which the training isnot yet complete.

Although some of various drawings illustrate a number of logical stagesin a particular order, stages which are not order dependent may bereordered and other stages may be combined or broken out. While somereordering or other groupings are specifically mentioned, others will beobvious to those of ordinary skill in the art, so the ordering andgroupings presented herein are not an exhaustive list of alternatives.Moreover, it should be recognized that the stages could be implementedin hardware, firmware, software or any combination thereof.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the scope of the claims to the precise forms disclosed. Manymodifications and variations are possible in view of the aboveteachings. The embodiments were chosen in order to best explain theprinciples underlying the claims and their practical applications, tothereby enable others skilled in the art to best use the embodimentswith various modifications as are suited to the particular usescontemplated.

What is claimed is:
 1. A method of performing automated objectinspection, comprising: at a computing device having one or moreprocessors and memory: obtaining a plurality of test images, each testimage capturing a respective one of a plurality of composite objectshave been assembled in accordance with a predefined common layout forthe plurality of composite objects, wherein: the predefined commonlayout specifies respective positions for two or more components of arespective composite object, and a predefined location and orientationfor a reference marker, in the respective composite object, and eachtest image captures respective portions of a corresponding compositeobject that contain the two or more components and the reference marker,and captures at least a portion of a surrounding environment of thecorresponding composite object, and; for each of the plurality of testimages: quantitatively determining a respective transformation from apredefined contour of the reference marker in the predefined commonlayout to a respective anchor contour corresponding to the referencemarker as captured in the test image; and applying an inverse of therespective transformation to at least a portion of the test image toobtain a respective regularized version of the test image, such that thereference markers captured in the respective regularized versions of theplurality of test images share an image-independent location andorientation; and performing independent automated inspection on each oftwo or more sub-portions of the respective regularized versions of thetest images, wherein each sub-portion of the respective regularizedversions of the test images corresponds to a respective one of the twoor more components of the respective composite object captured in thetest image.
 2. The method of claim 1, wherein quantitatively determininga respective transformation from a predefined contour of the referencemark in the predefined common layout to a respective anchor contourcorresponding to the reference mark as captured in the test imageincludes: extracting a plurality of contours from the test image;calculating a corresponding total area of each of the plurality ofcontours extracted from the test image; and identifying the respectiveanchor contour from among the plurality of contours extracted from thetest image in accordance with a determination that a difference betweena total area of the reference mark in the predefined common layout andthe corresponding total area of the anchor contour meets area-differencecriteria, wherein the area-difference criteria require that a differencein total area between two contours in comparison is below a thresholdvalue in order for the area-difference criteria to be met.
 3. The methodof claim 2, wherein identifying the respective anchor contour from amongthe plurality of contours extracted from the test image furtherincludes: identifying, from among the plurality of contours extractedfrom the test image, two or more candidate anchor contours that meet thearea-difference criteria; and selecting the anchor contour from the twoor more candidate contours based on relative closeness between therespective shape feature of each of the two or more candidate contoursand a shape feature of the reference mark in the predefined commonlayout.
 4. The method of claim 1, wherein quantitatively determining arespective transformation from a predefined contour of the referencemark in the predefined common layout to a respective anchor contourcorresponding to the reference mark as captured in the test imageincludes: after the respective anchor contour corresponding to thereference mark as captured in the test image is identified, obtaining anordered sequence of key coordinates for a minimum bounding rectangle ofthe respective anchor contour in the test image; obtaining an orderedsequence of key coordinates for a minimum bounding rectangle of thereference mark in the predefined common layout; and based on the orderedsequences of key points for the minimum bounding rectangles of thereference mark in the predefined common layout and the anchor contour inthe test image, determining a respective affine transformation from theminimum bounding rectangle of the reference mark in the predefinedcommon layout to the minimum bounding rectangle of the respective anchorcontour.
 5. The method of claim 4, wherein applying an inverse of therespective transformation to at least a portion of the test image toobtain a respective regularized version of the test image, such that thereference markers captured in the respective regularized versions of theplurality of test images share an image-independent location andorientation, includes: applying an inverse of the affine transformationto at least a portion of the test image, wherein the portion of the testimage includes the reference mark and the two or more components of thecomposite object.
 6. The method of claim 5, wherein performingindependent automated inspection on each of two or more sub-portions ofthe respective regularized versions of the test images includes: for therespective regularized version of each of the plurality of test images:extracting a first sub-portion of the respective regularized version ofthe test image at a location corresponding to a first component of thetwo or more components in accordance with a location of the firstcomponent as specified in the predefined common layout; extracting asecond sub-portion of the respective regularized version of the testimage at a location corresponding to a second component of the two ormore components in accordance with a location of the second component asspecified in the predefined common layout; and performing objectinspection on the first sub-portion of the respective regularizedversion of the test image based on a first model trained on trainingimages containing the first component and not containing the secondcomponent; and performing object inspection on the second sub-portion ofthe respective regularized version of the test image based on a secondmodel trained on training images containing the second component and notcontaining the first component.
 7. The method of claim 1, whereinquantitatively determining a respective transformation from a predefinedcontour of the reference mark in the predefined common layout to arespective anchor contour corresponding to the reference mark ascaptured in the test image includes: obtaining center coordinates of therespective anchor contour in the test image; calculating a translationoffset between center coordinates of the respective anchor contour inthe test image and center coordinates of the reference mark in thepredefined common layout; generating a first histogram Di(Theta′) ofdistances between the center coordinates of the respective anchorcontour and each of a sequence of n equally spaced points around therespective anchor contour; generating a second histogram Di (Theta) ofdistances between the center coordinates of the reference mark and eachof a sequence of n equally spaced points around a contour of thereference mark as specified in the predefined common layout; determiningan angular shift on the first histogram that will cause the histogram toachieve alignment with the second histogram; and using the translationoffset and the angular shift as the respective transformation from thepredefined contour of the reference mark in the predefined common layoutto the respective anchor contour corresponding to the reference mark ascaptured in the test image.
 8. A server system, comprising: one or moreprocessors; and memory storing one or more programs for execution by theone or more processors, the one or more programs including instructionsfor: obtaining a plurality of test images, each test image capturing arespective one of a plurality of composite objects have been assembledin accordance with a predefined common layout for the plurality ofcomposite objects, wherein: the predefined common layout specifiesrespective positions for two or more components of a respectivecomposite object, and a predefined location and orientation for areference marker, in the respective composite object, and each testimage captures respective portions of a corresponding composite objectthat contain the two or more components and the reference marker, andcaptures at least a portion of a surrounding environment of thecorresponding composite object, and; for each of the plurality of testimages: quantitatively determining a respective transformation from apredefined contour of the reference marker in the predefined commonlayout to a respective anchor contour corresponding to the referencemarker as captured in the test image; and applying an inverse of therespective transformation to at least a portion of the test image toobtain a respective regularized version of the test image, such that thereference markers captured in the respective regularized versions of theplurality of test images share an image-independent location andorientation; and performing independent automated inspection on each oftwo or more sub-portions of the respective regularized versions of thetest images, wherein each sub-portion of the respective regularizedversions of the test images corresponds to a respective one of the twoor more components of the respective composite object captured in thetest image.
 9. The server system of claim 8, wherein quantitativelydetermining a respective transformation from a predefined contour of thereference mark in the predefined common layout to a respective anchorcontour corresponding to the reference mark as captured in the testimage includes: extracting a plurality of contours from the test image;calculating a corresponding total area of each of the plurality ofcontours extracted from the test image; and identifying the respectiveanchor contour from among the plurality of contours extracted from thetest image in accordance with a determination that a difference betweena total area of the reference mark in the predefined common layout andthe corresponding total area of the anchor contour meets area-differencecriteria, wherein the area-difference criteria require that a differencein total area between two contours in comparison is below a thresholdvalue in order for the area-difference criteria to be met.
 10. Theserver system of claim 9, wherein quantitatively determining arespective transformation from a predefined contour of the referencemark in the predefined common layout to a respective anchor contourcorresponding to the reference mark as captured in the test imageincludes: identifying, from among the plurality of contours extractedfrom the test image, two or more candidate anchor contours that meet thearea-difference criteria; and selecting the anchor contour from the twoor more candidate contours based on relative closeness between therespective shape feature of each of the two or more candidate contoursand a shape feature of the reference mark in the predefined commonlayout.
 11. The server system of claim 8, wherein quantitativelydetermining a respective transformation from a predefined contour of thereference mark in the predefined common layout to a respective anchorcontour corresponding to the reference mark as captured in the testimage includes: after the respective anchor contour corresponding to thereference mark as captured in the test image is identified, obtaining anordered sequence of key coordinates for a minimum bounding rectangle ofthe respective anchor contour in the test image; obtaining an orderedsequence of key coordinates for a minimum bounding rectangle of thereference mark in the predefined common layout; and based on the orderedsequences of key points for the minimum bounding rectangles of thereference mark in the predefined common layout and the anchor contour inthe test image, determining a respective affine transformation from theminimum bounding rectangle of the reference mark in the predefinedcommon layout to the minimum bounding rectangle of the respective anchorcontour.
 12. The server system of claim 11, wherein applying an inverseof the respective transformation to at least a portion of the test imageto obtain a respective regularized version of the test image, such thatthe reference markers captured in the respective regularized versions ofthe plurality of test images share an image-independent location andorientation, includes: applying an inverse of the affine transformationto at least a portion of the test image, wherein the portion of the testimage includes the reference mark and the two or more components of thecomposite object.
 13. The server system of claim 12, wherein performingindependent automated inspection on each of two or more sub-portions ofthe respective regularized versions of the test images includes: for therespective regularized version of each of the plurality of test images:extracting a first sub-portion of the respective regularized version ofthe test image at a location corresponding to a first component of thetwo or more components in accordance with a location of the firstcomponent as specified in the predefined common layout; extracting asecond sub-portion of the respective regularized version of the testimage at a location corresponding to a second component of the two ormore components in accordance with a location of the second component asspecified in the predefined common layout; performing object inspectionon the first sub-portion of the respective regularized version of thetest image based on a first model trained on training images containingthe first component and not containing the second component; andperforming object inspection on the second sub-portion of the respectiveregularized version of the test image based on a second model trained ontraining images containing the second component and not containing thefirst component.
 14. The server system of claim 8, whereinquantitatively determining a respective transformation from a predefinedcontour of the reference mark in the predefined common layout to arespective anchor contour corresponding to the reference mark ascaptured in the test image includes: obtaining center coordinates of therespective anchor contour in the test image; calculating a translationoffset between center coordinates of the respective anchor contour inthe test image and center coordinates of the reference mark in thepredefined common layout; generating a first histogram Di(Theta′) ofdistances between the center coordinates of the respective anchorcontour and each of a sequence of n equally spaced points around therespective anchor contour; generating a second histogram Di (Theta) ofdistances between the center coordinates of the reference mark and eachof a sequence of n equally spaced points around a contour of thereference mark as specified in the predefined common layout; determiningan angular shift on the first histogram that will cause the histogram toachieve alignment with the second histogram; and using the translationoffset and the angular shift as the respective transformation from thepredefined contour of the reference mark in the predefined common layoutto the respective anchor contour corresponding to the reference mark ascaptured in the test image.
 15. A non-transitory computer-readablestorage medium, storing one or more programs configured for execution byone or more processors of a server system, the one or more programsincluding instructions for: obtaining a plurality of test images, eachtest image capturing a respective one of a plurality of compositeobjects have been assembled in accordance with a predefined commonlayout for the plurality of composite objects, wherein: the predefinedcommon layout specifies respective positions for two or more componentsof a respective composite object, and a predefined location andorientation for a reference marker, in the respective composite object,and each test image captures respective portions of a correspondingcomposite object that contain the two or more components and thereference marker, and captures at least a portion of a surroundingenvironment of the corresponding composite object, and; for each of theplurality of test images: quantitatively determining a respectivetransformation from a predefined contour of the reference marker in thepredefined common layout to a respective anchor contour corresponding tothe reference marker as captured in the test image; and applying aninverse of the respective transformation to at least a portion of thetest image to obtain a respective regularized version of the test image,such that the reference markers captured in the respective regularizedversions of the plurality of test images share an image-independentlocation and orientation; and performing independent automatedinspection on each of two or more sub-portions of the respectiveregularized versions of the test images, wherein each sub-portion of therespective regularized versions of the test images corresponds to arespective one of the two or more components of the respective compositeobject captured in the test image.
 16. The non-transitorycomputer-readable storage medium of claim 15, wherein quantitativelydetermining a respective transformation from a predefined contour of thereference mark in the predefined common layout to a respective anchorcontour corresponding to the reference mark as captured in the testimage includes: extracting a plurality of contours from the test image;calculating a corresponding total area of each of the plurality ofcontours extracted from the test image; and identifying the respectiveanchor contour from among the plurality of contours extracted from thetest image in accordance with a determination that a difference betweena total area of the reference mark in the predefined common layout andthe corresponding total area of the anchor contour meets area-differencecriteria, wherein the area-difference criteria require that a differencein total area between two contours in comparison is below a thresholdvalue in order for the area-difference criteria to be met.
 17. Thenon-transitory computer-readable storage medium of claim 16, whereinidentifying the respective anchor contour from among the plurality ofcontours extracted from the test image further includes: identifying,from among the plurality of contours extracted from the test image, twoor more candidate anchor contours that meet the area-differencecriteria; and selecting the anchor contour from the two or morecandidate contours based on relative closeness between the respectiveshape feature of each of the two or more candidate contours and a shapefeature of the reference mark in the predefined common layout.
 18. Thenon-transitory computer-readable storage medium of claim 15, whereinquantitatively determining a respective transformation from a predefinedcontour of the reference mark in the predefined common layout to arespective anchor contour corresponding to the reference mark ascaptured in the test image includes: after the respective anchor contourcorresponding to the reference mark as captured in the test image isidentified, obtaining an ordered sequence of key coordinates for aminimum bounding rectangle of the respective anchor contour in the testimage; obtaining an ordered sequence of key coordinates for a minimumbounding rectangle of the reference mark in the predefined commonlayout; and based on the ordered sequences of key points for the minimumbounding rectangles of the reference mark in the predefined commonlayout and the anchor contour in the test image, determining arespective affine transformation from the minimum bounding rectangle ofthe reference mark in the predefined common layout to the minimumbounding rectangle of the respective anchor contour.
 19. Thenon-transitory computer-readable storage medium of claim 18, whereinapplying an inverse of the respective transformation to at least aportion of the test image to obtain a respective regularized version ofthe test image, such that the reference markers captured in therespective regularized versions of the plurality of test images share animage-independent location and orientation, includes: applying aninverse of the affine transformation to at least a portion of the testimage, wherein the portion of the test image includes the reference markand the two or more components of the composite object.
 20. Thenon-transitory computer-readable storage medium of claim 19, whereinperforming independent automated inspection on each of two or moresub-portions of the respective regularized versions of the test imagesincludes: for the respective regularized version of each of theplurality of test images: extracting a first sub-portion of therespective regularized version of the test image at a locationcorresponding to a first component of the two or more components inaccordance with a location of the first component as specified in thepredefined common layout; extracting a second sub-portion of therespective regularized version of the test image at a locationcorresponding to a second component of the two or more components inaccordance with a location of the second component as specified in thepredefined common layout; and performing object inspection on the firstsub-portion of the respective regularized version of the test imagebased on a first model trained on training images containing the firstcomponent and not containing the second component; and performing objectinspection on the second sub-portion of the respective regularizedversion of the test image based on a second model trained on trainingimages containing the second component and not containing the firstcomponent.